Summary of problem:
When I convert raw data to a MrSID file there are many areas within the image that are the wrong color. Here is the raw image:
And here is the compressed MrSID:
What gives? Why are so many pixels missing?
Cause:
Some explanation of how GeoExpress and ArcMap handle nodata is in order. Although MrSID generation 3 images don’t have an alpha band, they do allow transparency settings at the time of encoding. You can either set transparency manually or let the metadata set it automatically if you do any sort of reprojection or rotation. When you rotate, reproject, mosaic, or crop an image the edges change and produce nodata values – the transparency settings have to be set to something since GeoExpress has to know what value to give to the empty “nodata” pixels. This value could be white, black, lime green, or any other value. Unfortunately that means that any other pixels in the image that legitimately have that value will also be set as transparent when a viewer opens the image. In the case of the image above the nodata value was set to the default value of black so all of the black pixels under the raised highway turned transparent. These pixels aren’t actually the wrong color – they are no color at all which allows the white background to show through.
Solution:
You have a couple solutions to this problem. If you use MG3 file format then set the transparency to the same color as the viewer that will eventually open these images. Most environments use white (255, 255, 255) but it depends on where the imagery will ultimately be used. In this scenario, those pixels will still be set as transparent, but it won’t matter because they will be the same color as the background. However it also allows the transparent pixels to show the value of the pixel underlying it if you have images that overlap one another. You also need to set the GeoExpress compression ratio to lossless compression so that the nodata pixels don’t get compressed to a value that is close to (but not quite) the nodata value. It also prevents the opposite (very dark pixels changing to the nodata value) which is what happened above. This is what the lossless MG3 looks like:
Alternately you can set the output to the newer MG4 format which creates a fourth band with alpha values. MG4 files don’t have any pixel value for RGB that is set as transparency – every value is valid. Instead, MG4s add an additional band (the Alpha band) that tells viewer software which pixels should be nodata pixels. LizardTech provides a free SDK for reading sid files (including MG4 support) so it’s hard for us to determine which of your applications support this newest file format. Please check with your software vendors first to make sure. If possible it’s highly recommended to use MG4 instead as this is the result:
If you go this route make sure to leave the alpha settings as opaque and GeoExpress will sort out the pixels on its own – if you set alpha values manually the result will be extra transparent pixels just as before. For a detailed explanation of why this happens be sure to read the related knowledgebase article:
A tale of two pixels – how nodata and transparency are handled in GeoExpress
Note: Any changes to the nodata and/or background value display settings in ArcMap will override the native settings of the MrSID image.
Additional:
Sometimes nodata values get out of hand in ArcMap. This often crops up when GeoExpress is set to automatically create aux.xml files when generating your sid images. If you generate a sid file without an aux.xml file, the nodata values embedded within the sid file are an intersection function – so only pixels with R=0 AND G=0 AND B=0 are transparent. However, for some reason, the aux.xml file treats them as a union – so pixels with R=0 OR G=0 OR B=0 are turned transparent. This means that any pixel that contains the nodata value inany single band will be transparent. Take a look:
Gross! There are even some transparent pixels in the street! However, the purpose of the aux.xml file is to override the embedded metadata in the file. This means it’s actually working as intended but not the way you may want it to. Both MG3 and MG4 files behave this same way if the aux.xml file is present. So essentially the solution to turn off aux.xml file creation altogether. The information in that file is already contained in the sid itself so it’s not really necessary. You can turn this off by going to Options > Preferences and unchecking the Generate box for “Aux XML Files” in the Georeferencing section. ArcMap will create its own aux.xml file when you use it to open the MrSID image, but the one ArcMap creates does not contain the transparency information so it can be safely ignored.