Effective Way To Read Images From A Csv File And Return A Tf.data.dataset Object

I have a csv file that contains two columns: the file path of the image which is stored as numpy arrays the label of the image Each row in the csv corresponds to one item (sample

Solution 1:

One way to do this is simply load those 2 files into variables and use tf.data.Dataset.from_tensor_slices (see https://www.tensorflow.org/guide/datasets#consuming_numpy_arrays)

Another way is to map the file path into dataset and do data pipelining to read and return it as (img, label) Here is the sample code from https://www.tensorflow.org/tutorials/load_data/images

defload_and_preprocess_image(path):
  image = tf.read_file(path)
  return preprocess_image(image)

ds = tf.data.Dataset.from_tensor_slices((all_image_paths, all_image_labels))

# The tuples are unpacked into the positional arguments of the mapped functiondefload_and_preprocess_from_path_label(path, label):
  return load_and_preprocess_image(path), label

image_label_ds = ds.map(load_and_preprocess_from_path_label)

Myself would prefer the second way if the data is too big for the memory, but the first one is handy for small data

Solution 2:

This tutorial should be a good place to start: https://www.tensorflow.org/tutorials/load_data/images

As explained in the link, load in your image paths and their labels. Create a dataset with from_tensor_slices of the paths and their labels, then map the paths (which are strings) with a preprocessing function to image Tensors.

ds = tf.data.Dataset.from_tensor_slices((all_image_paths, all_image_labels))

# The tuples are unpacked into the positional arguments of the mapped functiondefload_and_preprocess_from_path_label(path, label):
  return load_and_preprocess_image(path), label

image_label_ds = ds.map(load_and_preprocess_from_path_label)
image_label_ds

Follow the tutorial for step-by-step details. If your images are saved as numpy arrays rather than jpg files, you'll have to change some of the preprocessing but the overall flow should be very similar.