Flattening the Rx audio spectrum in WSJT-X

Bottom line (at the top)

For those who don't like to read much! For any digital node operating on any band under any conditions, using a reference spectrum to flatten the audio passband will, or at least can:

1. Make weak signals easier to see on the waterfall
2. Improve SNR accuracy/consistency for weak signals (and even stronger signals at times)
3. Improve weak signal decoding
4. Certainly do no harm, even if using a reference spectrum may not always help

Also important is that you should take a new reference spectrum if you make any changes to the receiver (e.g. gain settings, bandwidth settings, noise blanker). Also, don't use any form of digital noise reduction. It can badly distort decode SNR numbers and it can raise the decode threshold slightly so you may miss very weak signals. It's also very confusing for others when you report a 10GHz beacon reception SNR that's 7dB higher then it really is because you have ND reduction turned on and you don't tell people! Noise reduction does NOT, repeat NOT, improve decode sensitivity (in fact quite the opposite). The ability to decode and the SNR number generated by WSJTX are not functionally related.

The Details

Flattening the spectrum of the audio output from the receiver is important for three reasons. First, it gives a widegraph display on which weaker signals can be see. This is covered in detail at WSJT-X Wide Graph (waterfall) optimization. Second, a properly flattened audio spectrum permits the decoding of weaker signals under some conditions, especially when the signal covers most of the audio spectrum. For example those found when using Q65-60E for microwave EME operation. Third. a flat audio passband will give more accurate and consistent values for the SNR of decoded signals.

Most rigs, and especially older analog radios have an audio output passband that is anything but flat. The width is usually from somewhere around 300Hz to somewhere around 2500Hs with a peak around the center and some sort of asymmetric roll off at higher and lower frequencies. Basically a mess! Modern digitally based radios typically have pretty flat audio passbands with fairly sharp cutoff at high and low frequencies. However even digital based rigs with a fairly flat passband can sometimes benefit for further flattening of their audio spectrum. My IC-7100 with a direct USB connection to the internal soundcard has a pretty flat audio passband. Most of the time it needs no correction. However there are some circumstances (e.g. 10Ghz Q65-60E) where the use of a reference spectrum can improve the accuracy of decode SNR.

In this discussion I'm use the FT-897D as an example of an older, analog, rig, but much of the discussion applies to ANY rig of any age and type.

For the wide graph, where you are looking for digital signal traces on the waterfall, you ideally want a uniform background, so that weak signals stand out, wherever they are. WSJT-X provides two methods to do this, The first is via a "flatten" checkbox and the second is via a "Ref Spec" (reference spectrum) check box. Both will attempt to give a waterfall with a uniform background. "Flatten" is simple, fast, and required no additional operator input. "Ref Spec" gives better flattening, but required the user to collect and store a reference spectrum (see WSJT-X Wide Graph (waterfall) optimization for more details on that).

Briefly using the reference spectrum involves a few simple steps.

• Point the antenna at cold sky (away from the moon if there is a lot of activity on the band you are using). Make sure there are no traces on the waterfall. If there are birdies present, find another frequency which is clear from all signals.
• Set the RF and audio gain that you will normally be using, i.e. don't change anything from your normal operational conditions.
• If you have something like DNR (dynamic noise reduction) turn it off and never turn it on again for EME operation!
• From the "Tools" menus, click on "erase reference spectrum" (not sure this is needed, but it doesn't hurt)
• Uncheck "flatten" and "ref spec" boxes below the waterfall (again, not sure if this is really needed, but it doesn't hurt)
• Click of "measure reference spectrum"
• Wait at least 60s, then click on "Stop" (below the decode windows, next to "monitor)
• Click on "OK" in the windows that pops up to tell you the reference spectrum has been saved
• Check the "ref spec" box below the waterfall
• Observe the waterfall and spectrum displays. The waterfall should be uniform and the spectrum flat. If they are not, do it all again.

The flattened spectrum and smooth waterfall is not the end of the story. While both "flatten" and "ref spec" may sometimes give a similar looking Wide Graph, they are quite different when it come to the data which is actually passed along to the rest of the WSJT-X program for analysis and decoding. If the "flatten" box is checked (or even if it isn't), the raw audio data from the Rx is used for analysis by WSJT-X. On the other hand, if the "Ref Spec" box is checked, the flattened audio is sent to WSJT-X. The difference is illustrated below:

Here's the audio spectrum which is passed along to WSJT-X (and which is saved as a .wav file if desired). This is with "flatten checked ( or in deed with neither "flatten" nor "Ref Spec" checked):

For the FT-897D, this is the shape of the audio signal which WSJT-X looks at. The vertical scale for each plot is in dB. The -10dB points are round 580Hz and 2680Hz

Now, again with the FT-897D, here is the spectrum of the signal which is sent for analysis and decoding (and output as a .wav file) when "Ref Spec" is properly configured with a good reference spectrum

You can see that the passband is very flat. The usable passband is also probably wider then in the raw audio. It's about 350Hz to 2770Hz at the -0.5dB points

Now here are a series of screen-shots using the IC-7100:

This shows the audio directly from the IC-7100. It's pretty flat apart from a small rise at low frequencies. Normally this would not be a problem with most signals. However if using Q65-60E on 10Ghz with a sync tone at 700Hz (as is common practice), that small rise can distort SNR reports (but not decoding ability)

Here is what happens if you turn on "flatten". Looks awful. However my understanding is that the "flatten" option only affects the display, not the data which WSJTX analyzes internally

Now where is the spectrum after using a reference spectrum. It's even flatter then the original, plus the low frequency peak has gone. Also the flat region of the spectrum, is wider than that observed with just the raw signal.

Finally here's what the spectrum looks like with noise reduction turned on. The spectrum changes quickly with time as the DSP tries to analyze the noise and seperate it from signal (which in the case of weak digital signals, it can't see). To this is a current rather than cumulative spectrum. Peals at both ends plus random fluctuations over time in the mid region. For more on this see Noise Reduction

ONE (or several) NOTE(s) OF CAUTION about using the reference spectrum option is that if you change any receiver setting that affects the audio passband, you have to take a new reference spectrum. Some rigs allow the width of the Rx passband to be selected. My IC-7100 allows adjustable SSB passband (-6dB points) for example. Using the "flatten" option will do it's best to match whatever the passband shape is, but using the "reference spectrum" option requires the reference spectrum to be recoded at the same settings as are in being used at the time of use for best results. Also, whatever condition you use to record the reference spectrum should also be used for on-air decoding. Changing RF gain, for example, can slightly affect audio passband shape on some rigs. Similarly the use of any "tone controls" should be avoided, as should any significant change in audio signal level. In all cases any DSP functions should be turned off. Do NOT use things like DNR (Dynamic Noise Reduction). If you need a noise blanker you can use it, but record the reference spectrum with it turned on in that case. ALSO, when you have created the reference spectrum and activate it for the waterfall and spectrum display, make sure the spectrum display shows a FLAT spectrum and the waterfall is uniform from edge to edge. If it isn't, record the reference spectrum again and check again. Repeat if necessary until the spectrum is flat.

With some rigs, doing things like changing RF gain can affect the shape of the audio passband. It is always best to establish a new reference spectrum whenever you make any changes to the rig settings. So if you change the RF gain and turn on the noise blanker, or if you change the width of the audio passband, take a new reference spectrum.

Note also that if you save .wav files, using the reference spectrum will save them flattened, making further analysis of them easier. If you just use "flatten", the RAW, unflattened audio spectrum is saved. Also note that if your spectrum is already flat, using the "Flatten" option can make the spectrum worse. Look at the spectrum in the lower half of the waterfall, choosing "current" or "Cumulative" as the display option. Make sure whatever you are doing doesn't UN-flatten the spectrum!

Does this Matter

Yes, but probably not always or for everyone. The use of the simple "flatten" option has been shown to lead to decreased decode sensitivity for wide bandwidth signals on 10GHz, where the spreading can be quite wide, requiring large signal bandwidths with some modes. Whether or not there is any effect on decode sensitivity probably depends on the audio passband of the rig in question, where the "base" frequency (lowest tone) is positioned, and the total width of the signals being analyzed. For a mode like QRA65 and a high submode like D or E, the total signal width would be 877Hz and 1752Hz respectively (see red lines on the above spectral plots). It's fairly likely that variations of signal strength by 6-10dB which could well occur over a span of 1750Hz isn't going to help decoding! Indeed it has been seen to be an issue. On the other than, on HF and even the lower EME bands using A submodes, Modes like JT4 would have a bandwidth of <20Hz and JT65/QRA64 177/11 Hz, so a slope in the background probably wouldn't have a lot of effect. The closer the edge of the overall signal bandwidth get to the edge of the Rx audio bandpass, the worse the effect, because the roll-off at the edge of the bandpass is steepest.

Another effect of an unflattened spectrum can be incorrect signal reports, especially when the signal sync tone is near the lower end of the audio passband. In a worst case condition the SNR report can be 4 or 5dB off. If your passband is, say 400Hz to 2800Hz and the signal sync tone is at 500Hz or 600Hz, the SNR is very likely to be wrong.

I don't know for sure, but I suspect if you are using JT65A on 144 MHz, the shape of the audio spectrum probably doesn't matter much.

UPDATE 08/21 Looking at q65-60E signals on 10Ghz (Q65 has now replaced QRA64), it was again confirmed that the use of "flatten" rather then a reference spectrum does result in a slight reduction in decode sensitivity. It may only be 1dB or less, but at the decode margin you may see decodes when using the reference spectrum that you will miss if you are using "flatten". In addition, the reported signal strength of Q65-60E signals can also be affected by a spectrum which isn't flat, even on stronger signals. There may be several dB difference in reported signal strength between "flatten" and "reference". Possibly related to this is the observation that the selected sync tone frequency of Q65-60E signals on 10Ghz affects the reported SNR though there is little (if any?) change in decode sensitivity. If you put the Rx sysn tone at 600Hz, you may see the SNR reported by a decode to be up to 4dB stronger than it is with the sync tone at 1100Hz. At 1100Hz it's probably close to correct. At 600Hz the reported SNR is very optimistic. A referenced spectrum with the sync tone around 1000-1200Hz will probably give you the best estimate of signal strength for Q65-60E at the moment (WSJTX 2.5.0 -rc4).

Note that some of these reported SNR effects are probably due to the width of the signal for Q65-60E (1733Hz + spreading) and the position of the edges of the signal in relation to the available audio bandwidth.

Conclusion

For microwave EME, the use of a reference spectrum to flatten the audio, both for display and processing seems like a "no brainer". There's really no reason not to use it, and a number of reasons to use it. I don't know if there is any advantage (or disadvantage) to its use on the lower EME bands and over non-EME paths. It does make later analysis (using tools other than WSJT-X) of the saved .wav file somewhat easier. I've been using it for 1296 EME operation for several years with no problems (and nice, uniform, Wide Graph backgrounds).

There's a small question in my mind about Tx audio and whether it might have any effect of the transmission (and subsequent decoding) of wide bandwidth microwave EME signals. Analog based rigs may have a Tx audio bandwidth similar to their Rx audio bandwidth. It's not something I've ever tried to measure. I really don't know if a non-flat Tx audio bandwidth could result in problems, but it's certainly something to think about, especially for microwave EME with large signal bandwidths.

Bottom line (at the bottom)

For those who don't like to read much! For any digital node operating on any band under any conditions, useing a reference spectrum to flatten the audio passband will:

1. Make weak signals easier to see on the waterfall
2. Improve SNR accuracy/consistency for weak signals (and even strong signals under some circumstances)
3. Improve weak signal decoding (under some circumstances)
4. Will do no harm, even if it doesn't help under some conditions

Also important is that you should take a new reference spectrum if you make any changes to the receiver (e.g. gain settings). Also, don't use any form of digital noise reduction. It can badly distort decode SNR numbers and it can raise the decode threshold so you may miss very weak signals.

Acknowledgments

Thanks to Charlie, G3WDG, for useful discussions and to Bill, G4WJS, for useful insight into the internal workings of WSJT-X. However all and any technical error or lack of understanding of issues here are all mine!