os.walk not working as expected

This is the place for queries that don't fit in any of the other categories.

os.walk not working as expected

Postby always_stuck » Mon Nov 04, 2013 11:52 am

I'm trying to implement os.walk to list all files in a directory tree. When I run my code, I only get the files within the top level directory of my tree. I expected it to list all the files for all sub-directories in the entire. I have tried this with a number of different folders and always get the same result. Am I misunderstanding os.walk or is there another issue? I'm using windows 7 and python 2.5. Thanks

Code: Select all
for root, dirs, files in os.walk(os.path.join("\\\?\\I:\\Users\\MyWork\\")):
    for name in files:
        print name
always_stuck
 
Posts: 21
Joined: Fri Sep 20, 2013 10:47 am

Re: os.walk not working as expected

Postby Kebap » Mon Nov 04, 2013 12:48 pm

Hi al,

I have tried os.walk, and it worked as expected, listing files and folders also subfolders' contents. Please see my example in code:

Code: Select all
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> # I start in an empty folder:
>>> for i in os.walk("."):
...   print i
...
('.', [], [])
>>> # I create some test files and folders like so:
>>> #  .
>>> #  -- somefile.txt
>>> #  -- testfolder
>>> #  ---- anotherfile.txt
>>> os.mkdir("testfolder")
>>> open("somefile.txt", "a")
<open file 'somefile.txt', mode 'a' at 0x01D85390>
>>> open("testfolder/anotherfile.txt", "a")
<open file 'testfolder/anotherfile.txt', mode 'a' at 0x01DE9B78>
>>> # I test output of os.walk again:
>>> for i in os.walk("."):
...   print i
...
('.', ['testfolder'], ['somefile.txt'])
('.\\testfolder', [], ['anotherfile.txt'])
>>> # I test output of your function:
>>> for root, dirs, files in os.walk("."):
...   for name in files:
...     print name
...
somefile.txt
anotherfile.txt
>>>


I am not sure what you did wrong. As a suggestion, please try and use an easier function and earlier print for debugging.

always_stuck wrote:I'm trying to implement os.walk to list all files in a directory tree. When I run my code, I only get the files within the top level directory of my tree. I expected it to list all the files for all sub-directories in the entire. I have tried this with a number of different folders and always get the same result. Am I misunderstanding os.walk or is there another issue? I'm using windows 7 and python 2.5. Thanks

Code: Select all
for root, dirs, files in os.walk(os.path.join("\\\?\\I:\\Users\\MyWork\\")):
    for name in files:
        print name
Learn: How To Ask Questions The Smart Way
Join the #python-forum IRC channel on irc.freenode.net and chat with uns directly!
Kebap
 
Posts: 400
Joined: Thu Apr 04, 2013 1:17 pm
Location: Germany, Europe

Re: os.walk not working as expected

Postby always_stuck » Mon Nov 04, 2013 1:05 pm

Kebap, what version of python and what o.s are you using? When I try to reproduce your test, nothing is returned...
always_stuck
 
Posts: 21
Joined: Fri Sep 20, 2013 10:47 am

Re: os.walk not working as expected

Postby Kebap » Mon Nov 04, 2013 1:12 pm

Python 2.7.2 on Win 7

Please post your complete test for review.
Learn: How To Ask Questions The Smart Way
Join the #python-forum IRC channel on irc.freenode.net and chat with uns directly!
Kebap
 
Posts: 400
Joined: Thu Apr 04, 2013 1:17 pm
Location: Germany, Europe

Re: os.walk not working as expected

Postby always_stuck » Mon Nov 04, 2013 1:55 pm

My complete test is the same as yours!

Created folder 'test' on desktop. Added file 'test1.txt' and sub-directory 'test2'. Within 'test2' folder I added another file 'test2.txt'. Results below.

Code: Select all
>>> for i in os.walk('c:\Users\R924073\Desktop\test\\'):
       print i

   
>>>


Was about to post this but then noticed it wasn't returning anything because I hadn't escaped the special characters properly. Stupid me! Your example works now.

However, when I try to run the test with the real folder location I want to use, os.walk only prints the folders and files within the top directory, and 1 level of its sub-directories. This is where I'm stuck. The top directory of my tree is 'I:\\RND-ILLINOIS\\PROJECTS-US\\OFFLINE-DEVELOPMENT-US\\4-TURTLE-KEEP'

All special characters have been escaped, so I'm not sure what else might be causing os.walk to stop after level 1 of the tree...
always_stuck
 
Posts: 21
Joined: Fri Sep 20, 2013 10:47 am

Re: os.walk not working as expected

Postby Kebap » Mon Nov 04, 2013 2:54 pm

Please create for test a sub-folder in the location 'I:\\RND-ILLINOIS\\PROJECTS-US\\OFFLINE-DEVELOPMENT-US\\4-TURTLE-KEEP'. Just like you created on desktop. Then see if this test-folder is handled correctly by os.walk(). Maybe there is a problem with the other folders. For example, symbolic links won't work automagically.
Learn: How To Ask Questions The Smart Way
Join the #python-forum IRC channel on irc.freenode.net and chat with uns directly!
Kebap
 
Posts: 400
Joined: Thu Apr 04, 2013 1:17 pm
Location: Germany, Europe

Re: os.walk not working as expected

Postby always_stuck » Mon Nov 04, 2013 4:03 pm

Kebap wrote:Please create for test a sub-folder in the location 'I:\\RND-ILLINOIS\\PROJECTS-US\\OFFLINE-DEVELOPMENT-US\\4-TURTLE-KEEP'. Just like you created on desktop. Then see if this test-folder is handled correctly by os.walk(). Maybe there is a problem with the other folders. For example, symbolic links won't work automagically.


Just tried that, same result. No directories or files were found within the 'test' folder in the location 'I:\\RND-ILLINOIS\\PROJECTS-US\\OFFLINE-DEVELOPMENT-US\\4-TURTLE-KEEP' despite the same ones existing as previously.
always_stuck
 
Posts: 21
Joined: Fri Sep 20, 2013 10:47 am

Re: os.walk not working as expected

Postby always_stuck » Mon Nov 04, 2013 4:11 pm

The problem seems to be that os.walk only returns the sub-directories and files of a folder if that folder ends in '\\'. This is fine for the top of the directory tree where I manually add '\\' to the end of the foldername passed to os.walk, but for the sub-directories returned by os.walk there is no final '\\'. Is this a common issue with os.walk or is this still something to do with my directory tree structure/access?
always_stuck
 
Posts: 21
Joined: Fri Sep 20, 2013 10:47 am

Re: os.walk not working as expected

Postby ochichinyezaboombwa » Wed Nov 06, 2013 3:56 am

There is no magic, there probably is some typo somewhere. Try this:
Code: Select all
import glob, os
folder = "I:\\RND-ILLINOIS\\PROJECTS-US\\OFFLINE-DEVELOPMENT-US\\4-TURTLE-KEEP"
assert os.path.isdir(folder)
mask = os.path.join(folder, "*")
print glob.glob(mask)


Alternatively,
Code: Select all
dir I:\RND-ILLINOIS\PROJECTS-US\OFFLINE-DEVELOPMENT-US\4-TURTLE-KEEP
at command prompt. (Is it still "dir" under Windows?).
ochichinyezaboombwa
 
Posts: 200
Joined: Tue Jun 04, 2013 7:53 pm


Return to General Coding Help

Who is online

Users browsing this forum: Google [Bot] and 4 guests

cron