1. Almost all American popular music (except some jazz) is based on the Major Diatonic Scale.
2. Every song is written in a key, which is defined by the notes that are natural, sharp or flat in that key. You can find the notes in any key by applying The Formula of construction, which identifies were the whole steps and half steps occur between sequential notes.
3. From that Major Diatonic Scale we come up with a set of seven chords, or triads, by stacking a root, 3rd and 5th. Those basic chords will be Major, minor or diminished, and the order in which they occur transfers to every Major key.
4. It is likely that one of those chords (in a given key) will be "right" in a song you are trying to figure out by ear. (Note: Like everything in music, there are exceptions - but you can be pretty sure most of those chords will work with *most* popular songs)
Now a couple of interesting exceptions and tricks to use to hone in on even more accurate playing by ear.
First of all, what about that weird sounding diminished chord that is the naturally occurring VII chord in the scale line triads? "I hardly ever see songs that call for one," you say. "Besides, they sound kind of strange compared to the majors and minors in the scale line. Not wrong, exactly, just....strange."
Right you are. Diminished chords have many uses and are often used in jazz, both in scale lines and as transitional chords between the scale line ones. But in popular, folk, blues and country they are hardly ever used. However, songwriters still want to use the VII chord somehow. So what they do is very simple. They LOWER the root of the naturally occurring VII chord by 1/2 step and look what happens:
VII chord in the key of G Major: F#dim : C
> 1 1/2 steps
A
> 1 1/2 steps
F#
Now, with root lowered 1/2 step:
C
> 1 1/2 steps
A
> 2 steps
F
So what do we now have? An F Major chord! Instead of the weird sounding diminished VII chord (which was "correct" if we stick with only the notes in the key of G Major), we now have a wonderful sounding straight Major chord, one that sounds just a tiny bit out of the key due to the use of the F natural as the root, but still includes two other notes that are IN the key.
This is know as the "Flat VII chord" in a scale line. You can do the same thing with any of the keys: lower the root of the VII chord (diminished) by 1/2 step and you will get a Major chord. One of the most common progressions in popular music is the I - bVII - IV. Songwriters like James Taylor use it all the time ("Fire and Rain" "Country Road" "Machine Gun Kelly" and many others). Along with the I - IV - V progression, it is the most common grouping of chords in most forms of American popular music. Try it in a few keys! You will almost immediately be reminded of many, many songs you've heard.
This insertion of a new note leads to another question: Can I alter other chords in the scale line to develop interesting sounding chords? Well, of course! Just remember though that the more you insert notes outside the key you're playing into the scale line chords, the more "outside" they will sound. This can be a good thing from the perspective of making interesting sounding music. Let's face it, playing ONLY chords in the scale line in a key can sound kind of plain. But keep in mind the fact that most popular songs need to be somewhat logical sounding and centered. Use altered chords with caution.
So where do I start when I want to figure out a song by ear? Keeping in mind that most songs begin with the I chord (the chord that is the root or first chord of the key) I listen not to guitar part - but rather I listen to the bass! Bass players almost always play the roots of the chords that are being played by the guitarist or keyboard player, at least at the beginning of a measure or when a chord change happens. I pick my low E string, move up the neck and try to match (albeit, one octave higher) the note the bass player is playing on the first downbeat of the song, or where the vocals come in on a down beat. That little trick almost always works to help me identify which key the song is in. Then it's just a matter of thinking about the other scale-line triads available to me in the key and try to use one of them when I hear a change.
This is not an exact science of course and it takes some time and experience to be reasonably accurate. Also, there will be times when TWO different chords sound OK. You then have to go with one for a while and decide in the long term whether or not you're pleased with the sound. Keep in mind that chord changes usually happen on the first beat of a measure, or half way through the measure.
So there you have it - a very basic overview of understanding what it is to play in a Major key, the chords that are developed from that key, and how to make an educated guess as to where they will go in the music. There is a whole world of music beyond this of course, beginning with the more complicated 4 and 5-tone chords. Explore those as you become comfortable with the concepts I've laid out.
Good luck with your quest! Figuring out songs by ear is fun and will definitely make you a better player.
Peace & good music,
Gene