Steps_per_epoch And Validation_steps For Infinite Dataset In Keras Model

I have a huge dataset of csv files having a volume of around 200GB. I don't know the total number of records in the dataset. I'm using make_csv_dataset to create a PreFetchDataset

Solution 1:

I see no other option than iterating through your entire dataset.

ds = tf.data.experimental.make_csv_dataset('myfile.csv', batch_size=16, num_epochs=1)

for ix, _ inenumerate(ds, 1):
    passprint('The total number of steps is', ix)

Don't forget the num_epochs argument.