tensorflow中next_batch的具体使用

linux  /  管理员 发布于 7年前   296

本文介绍了tensorflow中next_batch的具体使用，分享给大家，具体如下：

此处给出了几种不同的next_batch方法，该文章只是做出代码片段的解释，以备以后查看：

 def next_batch(self, batch_size, fake_data=False):  """Return the next `batch_size` examples from this data set."""  if fake_data:   fake_image = [1] * 784   if self.one_hot:    fake_label = [1] + [0] * 9   else:    fake_label = 0   return [fake_image for _ in xrange(batch_size)], [     fake_label for _ in xrange(batch_size)   ]  start = self._index_in_epoch  self._index_in_epoch += batch_size  if self._index_in_epoch > self._num_examples: # epoch中的句子下标是否大于所有语料的个数，如果为True,开始新一轮的遍历   # Finished epoch   self._epochs_completed += 1   # Shuffle the data   perm = numpy.arange(self._num_examples) # arange函数用于创建等差数组   numpy.random.shuffle(perm) # 打乱   self._images = self._images[perm]   self._labels = self._labels[perm]   # Start next epoch   start = 0   self._index_in_epoch = batch_size   assert batch_size <= self._num_examples  end = self._index_in_epoch  return self._images[start:end], self._labels[start:end]

该段代码摘自mnist.py文件，从代码第12行start = self._index_in_epoch开始解释，_index_in_epoch-1是上一次batch个图片中最后一张图片的下边，这次epoch第一张图片的下标是从 _index_in_epoch开始，最后一张图片的下标是_index_in_epoch+batch, 如果 _index_in_epoch 大于语料中图片的个数，表示这个epoch是不合适的，就算是完成了语料的一遍的遍历，所以应该对图片洗牌然后开始新一轮的语料组成batch开始

def ptb_iterator(raw_data, batch_size, num_steps): """Iterate on the raw PTB data. This generates batch_size pointers into the raw PTB data, and allows minibatch iteration along these pointers. Args:  raw_data: one of the raw data outputs from ptb_raw_data.  batch_size: int, the batch size.  num_steps: int, the number of unrolls. Yields:  Pairs of the batched data, each a matrix of shape [batch_size, num_steps].  The second element of the tuple is the same data time-shifted to the  right by one. Raises:  ValueError: if batch_size or num_steps are too high. """ raw_data = np.array(raw_data, dtype=np.int32) data_len = len(raw_data) batch_len = data_len // batch_size #有多少个batch data = np.zeros([batch_size, batch_len], dtype=np.int32) # batch_len 有多少个单词 for i in range(batch_size): # batch_size 有多少个batch  data[i] = raw_data[batch_len * i:batch_len * (i + 1)] epoch_size = (batch_len - 1) // num_steps # batch_len 是指一个batch中有多少个句子 #epoch_size = ((len(data) // model.batch_size) - 1) // model.num_steps # // 表示整数除法 if epoch_size == 0:  raise ValueError("epoch_size == 0, decrease batch_size or num_steps") for i in range(epoch_size):  x = data[:, i*num_steps:(i+1)*num_steps]  y = data[:, i*num_steps+1:(i+1)*num_steps+1]  yield (x, y)

第三种方式：

  def next(self, batch_size):    """ Return a batch of data. When dataset end is reached, start over.    """    if self.batch_id == len(self.data):      self.batch_id = 0    batch_data = (self.data[self.batch_id:min(self.batch_id + batch_size, len(self.data))])    batch_labels = (self.labels[self.batch_id:min(self.batch_id + batch_size, len(self.data))])    batch_seqlen = (self.seqlen[self.batch_id:min(self.batch_id + batch_size, len(self.data))])    self.batch_id = min(self.batch_id + batch_size, len(self.data))    return batch_data, batch_labels, batch_seqlen

第四种方式：

def batch_iter(sourceData, batch_size, num_epochs, shuffle=True):  data = np.array(sourceData) # 将sourceData转换为array存储  data_size = len(sourceData)  num_batches_per_epoch = int(len(sourceData) / batch_size) + 1  for epoch in range(num_epochs):    # Shuffle the data at each epoch    if shuffle:      shuffle_indices = np.random.permutation(np.arange(data_size))      shuffled_data = sourceData[shuffle_indices]    else:      shuffled_data = sourceData    for batch_num in range(num_batches_per_epoch):      start_index = batch_num * batch_size      end_index = min((batch_num + 1) * batch_size, data_size)      yield shuffled_data[start_index:end_index]

迭代器的用法，具体学习Python迭代器的用法

另外需要注意的是，前三种方式只是所有语料遍历一次，而最后一种方法是，所有语料遍历了num_epochs次

以上就是本文的全部内容，希望对大家的学习有所帮助，也希望大家多多支持。