-9999 As Missing Value With Numpy.genfromtxt()

Lets say I have a dumb text file with the contents: Year Recon Observed 1505 162.38 23 1506 46.14 -9999 1507 147.49 -9999 -9999 is

Solution 1:

Nope, you're not doing anything wrong. Using the missing_values argument indeed tells np.genfromtxt that the corresponding values should be flagged as "missing/invalid". The problem is that dealing with missing values is only supported if you use the usemask=True argument (I probably should have made that clearer in the documentation, my bad).

With usemask=True, the output is a masked array. You can transform it into a regular ndarray with the missing values replaced by np.nan with the method .filled(np.nan).

Be careful, though: if you have column that was detected as having a int dtype and you try to fill its missing values with np.nan, you won't get what you expect (np.nan is only supported for float columns).

Solution 2:

Trying:

>>> x = np.genfromtxt("file.txt",names = True, missing_values = "-9999", dtype=None)
>>> x
array([(1505, 162.38, 23), (1506, 46.14, -9999), (1507, 147.49, -9999)], 
      dtype=[('Year', '<i8'), ('Recon', '<f8'), ('Observed', '<i8')])

does not give the correct answer. So just making it a string doesn't help. However, if an additional flag, usemask=True is added, you get:

>>> x = np.genfromtxt("file.txt",names = True, missing_values = -9999, dtype=None, usemask=True)
>>> x
masked_array(data = [(1505, 162.38, 23) (1506, 46.14, --) (1507, 147.49, --)],
             mask = [(False, False, False) (False, False, True) (False, False, True)],
       fill_value = (999999, 1e+20, 999999),
            dtype = [('Year', '<i8'), ('Recon', '<f8'), ('Observed', '<i8')])  

which gives what you want in a MaskedArray, which may be useable for you anyway.

Solution 3:

The numpy documentation at SciPy suggests that the missing_value should be a string to work the way you want. A straight numeric value seems to be interpreted as a column index.