This post is specifically about submitting to literary magazines, so some of this would not be applicable for genre markets where things are a little different.
Know the Markets: This seems obvious but apparently it isn't because I've heard multiple editors say they get submissions they wouldn't even consider (ie, poetry at a journal that doesn't publish poetry, a genre horror story sent to an uber high brow literary magazine.) So assume you're smart enough to not do that. Beyond this, there are different aesthetics to different magazines. You should have a sense of the differences between markets by reading the magazines. You can subscribe, read content online, or just go to the library and page through an issue or two. Some markets take stuff that is experimental, some take stuff that is more pop-culture friendly, some only take work with a specific theme. For example, I have a few stories that are kind of funny. I never see stories like these in what I think of the more old fashioned literary magazines (think university funded, white cover, with a picture of abstract art on the cover.) Some of the more quirky magazines do take stuff like this. I write some stories the straddle the line between literary and sci fi-- just because sci fi is in the equation that doesn't mean they're a good fit for Fantasy & Science Fiction. I think over time I've gotten to high rates of positive rejections because I have a better sense of the market--this way you're not wasting your own time, or the editor's.
Submit simultaneously (sending the same story to multiple markets at the same time) when you can--don't when you shouldn't: I can only think of a handful of literary markets that don't take SSs--and some of them are very good--but weigh your options. If you're planning on pocketing a few publications while you finish your novel so you can possibly have some pubs when you start querying agents, you might want to avoid those magazines. (if the probability of being rejected is still 95%, there's no point spending extra time just to get rejected). But if you have time, or if there is something specifically desirable about that market, you could wait it out (as I mentioned in a previous post, if you're trying to place a very long story, you have fewer markets, so you might not have the luxury of avoiding no-SS magazines). If you submit to a magazine simultaneously when you're not supposed to and you get caught, you can get blacklisted from that magazine. There's a fair probability that if you did it you wouldn't get caught, but if you did, consider that the literary world is small. I know someone this happened to and they got an irate letter (it might have been a phone call??) from an editor.
Tier markets: When thinking about what type of story you have, start putting together a tiered list of desired markets. You don't want to submit to Best Market on Earth and Meh Market at the same time and have the latter get back to you first. (Standard practice is to go with whichever market gets back to your first. You can't say "I'm waiting to hear back from Magazine X," -- you're going to come off like someone who doesn't know anything about the business and that editor would hate you.) Don't submit anywhere where you wouldn't, on second thought, want to be published. It's sometimes hard to do the calculus about where the top tier should be. You can submit to places that are extremely prestigious and hard to get into (<1% acceptance rates), but do so only if you think your story is that good, AND with the knowledge that it's going to take time waiting to hear back. In other words, if you're out of your league you may be waiting a really long time to get a rejection from a magazine you had no hope of getting into. Not a big deal if it's a mag that takes two months to get back to you. Definitely a big deal if they take a year (not naming names..;)
A note about being timid for people who are timid: there are some people who won't submit somewhere because they assume their work isn't good enough. If you have enough feedback to suggest you are a good writer (feedback from people who are not loved ones), don't assume you aren't good enough for a competitive market. Back when I first started submitting in college, this was when all submissions were via snail mail, and if there was a substantial web presence of magazines on the internet, I had no idea of it. I certainly didn't know any other writers. If I had known that it's really desirable and hard to get a piece in Glimmer Train, I would have never submitted there. Glimmer Train was my first acceptance--I had no idea I was even in the ballpark because I only knew the magazine from actually reading it, not hearing anyone talk about it or reading statistics. The stories seemed accessible, there was a train on the cover, and from reading their rather generous "About the authors" section, it was clear they took emerging (read: never published before) authors. So on the one hand you have clueless, terrible writers who think they are amazing, and on the other you have good authors who assume they are bad. You need to have a good sense of what league each of your stories is in.
How many simultaneous submissions do you send?: My standard was to have one story out at three markets at a time. Then I went to a writing conference where there was a roundtable of editors who said do five to ten markets at a time for a single story. That astounded me. I think ten is excessive because most people don't know enough markets where you would be targeting submissions well and you could be burning potential markets if you have multiple stories out at the same time. (Some markets only want you to submit once a submission period and will notice if you do more.) I settled on five and that seemed to work well from me when I was submitting aggressively. When I got a rejection I would send out to another market so I always had five active submissions per story. Now that I'm less focused on publishing stories, I'm not really doing five at a time, and I'm more patient about waiting for some markets.
Stagger fast and slow markets: Say you're sending out a story to five markets. I like to have some of these be markets that get back to you more quickly (3 months or under, say) and to mix those in with those that take longer. Sometimes you get feedback (of some sort) from the faster markets that would influence your decision making, and also I like to keep things rolling. This gives you some time to research potential other markets.
Calibrate based on feedback: If you sent a story to five different markets and got back five form rejections, and no positive rejections, you might want to think about if the story isn't working. (This assumes I'm not talking about five markets that are impossible to get into, in which case rejection isn't a reflection of how good you are.) A positive rejections is anything that says "No thanks on this, but please submit to us again/ we'd like to see more from you/ send to us at any time." Any positive personal note is a positive rejection. I can't tell you how many times I have heard people say "well I didn't think they meant it" while also hearing editors say "YES WE MEANT IT." Assume that a welcome mat is a welcome mat. So anyway, when I'm getting positive rejections, I keep sending to other markets without changing anything about the story. I'm not a big believer in editing something when I don't have a secret gut feeling telling me there's something wrong with it.
Keep meticulous records: I use Duotrope, which you don't have to, but by itself it isn't enough. I have an Excel file where for each story it will have every market it has been submitted to, when it was sent and when it was rejected, what kind of rejection it was, and the ratio of positive rejections to overall number of submissions. This is helpful for a variety of reasons: it's essentially a ledger of all your relationships with different magazines. Have another tab in the worksheet where you keep a dated list of all magazines who have sent you a positive rejection. These are good markers to hit again, but don't overstay your invitation. Unless you have specifically heard otherwise, I don't think you should submit more than once in the same submission period to the same magazine. An exception would be if the same magazine opens a themed issue or you received particularly encouraging notes from the editor.
When to nudge: By nudge I mean send a "Hello, it's been several months and you have not responded about my submission." Don't do this unless 1) the magazine says on their website to nudge if you haven't heard in X amount of time or 2) an excessive amount of time has gone by with no response (and maybe you have info that other people have gotten their rejections back). What is excessive? One month of waiting is not, neither is six. If you're on Duotrope you have some sense of what is an abnormal amount of time for that market--it might be the case that your submission was lost, or it might be that you were shortlisted and they just haven't told you.
Take what editors say at face value: I'm repeating myself, but if they said they'd like to hear from you again, they aren't kidding. They have specific form rejections that say "Thank you for your submission but no" and form or personalized rejections that say "try us again." They mean it. If you meet an editor or someone who works at the magazine who expresses an interest in your work, assume that they are not just being polite.
Things that should be in your cover letter: I think people stress about this too much. Bare bones is fine. Don't include anything cute but don't leave out stuff relevant to forming an advantage.
Dear [Editor/s / Editor's name],
I am submitted a short story, "Storyface," (1,200 words) for consideration at Bumbleboo Magazine. [If relevant include any of the following: You welcomed additional submissions from me last year after I reading a different story, "Whatever." / A previous story of mine, "Whatever" made your shortlist last year / I spoke with you briefly at AWP and you encouraged me to submit. / This story was an honorable mention/finalist in the Some Story Contest in 2017 {obviously this is only the case when you received an honorable mention or finaled, but were not published by the other magazine} it is okay if you have nothing to put as a second sentence in this paragraph-- plenty of stories get picked up from no where]
My fiction has also appeared in Magazine X, Magazine Y, and Magazine Z. (list in order of prestige, not date of publication. listing more than 3 or 4 starts to get weird). My essays/nonfiction have also appeared in Magazine Q. (include relevant publications, leave off academic publications unless the content is related, leave off blog posts unless you run a very popular blog. If you have received honorable mentions or finaled in contests for other stories than the one you're submitting that did not result in publication, mention these. If you have an MFA or have been to a juried workshop, you can mention it, although this is a lot less relevant. If you have no other publications it is fine to have just one paragraph and maybe throw in a line of bio. Don't sweat this--everyone who ever goes on to get published at some point did not have publications). Thank you for your consideration.
Name
Nothing cute. Nothing about your family or adorable pets. nothing about the themes of your work, a summary of your work, or what you think it is about -- let the thing speak for itself. From reading various blogs it appears that business correspondence often stresses people out particularly if they are detail oriented. It's not a query letter for a novel, it's a business letter-- it's okay to keep it short and simple.
Submission fees: There are strong feelings about whether or not literary magazines should charge submission fees. (if you're not familiar with the debate, this and this show the general debate). There are markets that charge them and markets that don't, so if you really don't want to pay them, you don't have to. Some people think the fees are outrageous because writers often don't have much money, and if you send a story 5 places, that could be $15. Say you have seven stories-- that can add up. The part where I agree with magazines that charge fees: 1) it is about the same amount as postage used to be back in snail mail days 2) most of these magazines have people who either work for free or very little 3) there are way more writers that want to get published than there are writers who want to get published who support other writers by purchasing magazine subscriptions. With some exceptions, literary magazines are not money-making schemes. They are a niche market with not enough people support that market. Although it is interesting: consider a comparison to some of the top sci fi magazines: no submission fees, they pay professional rates, and turnaround time for submissions is ridiculously fast (at Clarksworld, something like 0 to 7 days). At that particular magazine, the readers are remote volunteers who pass good stuff up to senior editors who give a second pass. How is that so normal for scifi, but so uncommon for literary fiction? I don't know the whole story (and I'm super curious), but I think a main reason is that sci fi (and other genres like romance) have really really, strong, loyal fanbases. I think your average romance fan buys, consumes, and supports romance writing far more than a specific subset of literary writers who are more or less only interested in their own writing.
One thing about fees: a submission fee of more than 4 dollars is unusual and should be avoided. Contest fees make sense to be higher, but submission fees themselves don't need to be that high.
Contests: Three things about contests: assume you are not going to win them, the entry fee should make sense, and time when you enter them intelligently. Maybe I'm wrong, but I treat contest submissions as slightly less viable than regular submissions. (You're also a lot less likely to get personalized notes for contest submissions). So don't spend a lot of time waiting to hear back from a contest--submit to multiple magazines and assume that you won't win, unless there is some extenuating circumstance. A decade ago, entry fees used to be 10 dollars, maybe 15, but they really have been creeping up lately. I almost never see 10 dollar ones, and rarely 15. 20 is pretty standard, but do the math: if they are charging 20 per entry, but only offer a 200 dollar prize, something is fishy. A 30 dollar submission fee for a short story or series of poems seems excessive (unless the prize is enormous and maybe they offer 1st through 3rd place prizes)-- however, that number is pretty normal for a contest submission of a novel, short story collection, or full-sized poetry chapbook. Don't enter a story that's been rejected by that same magazine already (even if it's been revised). Lastly, if you enter a contest with an entry fee, do so the last week before the deadline. If you've been sending that story to multiple markets, there's a chance you might get an acceptance from another magazine after paying the entry fee to the contest market, so basically you just wasted your money (because you will have to withdraw from the contest with no refund). And given how long the time lag is between when contests open and when they close for submissions, there's some chance you might want to change something in your work during that period of time. There's no benefit to submitting early (or at least one I can see).
How long does a "try us again" invitation last? I include this because I found out that I'm not the only person that anxiously worries about this: Say you get a positive rejection from Market A. You submit to them again the following submission period (say 6 months or a year later) and you get a form rejection that doesn't specifically ask you to submit again. Does this mean you've been uninvited from being invited to submit again? I think it's okay to use the positive rejection a couple more times, but if you keep getting several consecutive form rejections in a row, you might want to rethink of the market's even a good match for you. (Also there are enough markets out there that you don't need to hit the same market over and over.) Editors do remember people in both positive and negative ways. I don't think it would be strange for you to submit to the same market once a year if you aren't get straight form rejections over and over--that said, there's enough markets out there that you should be able to cast a broader net.
If you get an acceptance: First re-confirm that the market is definitely taking the piece. Sometimes the editor will ask "is this story still available?" or say something like "We love Story X and would like to put it in Issue 47" in their first email to you. I don't consider it formally spoken for until I've emailed them back and they've confirmed with me. (It's never been an issue, but just in case the editor flakes out). Once you've confirmed, either use Submittable or email to contact the other editors to withdraw from the other markets the story is also at. Just say that you're withdrawing because it has been accepted elsewhere, be polite, and that's it. You're a dick if you forget to do this. If there's a contract to sign, (often there isn't if payment isn't involved), make sure you read it. There are several decent places that describe the typical rights discussed.
Factor the Summer in: This only matters if you are concerned about getting publications quicker, but realize that a lot of magazines are university-affiliated and shut down over the summer. There are markets that are open during the summer, but there are definitely fewer. Make sure to submit handsomely in the spring because if you keep putting it off and then all the sudden it's June, you may have limited options for several months.
Don't edit while submitting: Submittable now lets you do this, but you shouldn't. I have used Submittable both as a writer and an editor--you can't always tell when editors are actually looking at your piece (sometimes it doesn't say "In Progress" when it actually is). You should be sure it's done before sending it out.
Don't respond to rejections: Not politely or rudely. You get nothing from either (well, you get a reputation for the latter). Editors are not running an advice service--they don't owe you feedback.
Throwing in the towel: When should you give up on a story that you keep sending out but no one is taking? If you keep getting positive rejections, I would keep sending it out. If you are tired of the rejections, put the story away for a while and then read it with fresh eyes. If you still believe in it, keep sending it out. Publishing is really subjective--getting a rejection doesn't mean it's a bad story.
Happy submitting!