Best way to bring a list in from a config file

This is the place for queries that don't fit in any of the other categories.

Best way to bring a list in from a config file

Postby pnelsonsr » Mon Jan 27, 2014 11:43 pm

I have an application that pulls in var=val lines from a config file (app.cfg kind of thing). I generally have been using just a string like:
Code: Select all
files='..mylib/mylib.py applib.py app.py'

An pulling this into a dictionary within a running app by opening the config file and reading each line and then splitting the line at = and putting the 'files' part as the key and the rest as the value. This looks something like:
Code: Select all
CFGVALUES={}
with open(CFGFILE,'r') as f :
  for line in f :
    if not line.startswith('#') and line.strip() :
      CFGVALUES[line.strip().split('=')[0]]=line.strip().split('=')[1].replace('\'','')

This works fine but what I would really like to do is use a list in the config file like:
Code: Select all
files=['..mylib/mylib.py', 'applib.py', 'app.py']

This took me a while to get working so now I'm not sure if the way I got it working is really the best way. But I was un-successful at doing it any other way. Ultimately I would like the outcome to be something like:
Code: Select all
print CFGVALUES
{'files': ['..mylib/mylib.py', 'applib.py', 'app.py']}

The only way I could get this output was doing something like:
Code: Select all
val=''
with open(CFGFILE,'r') as f :
  for line in f :
    if not line.startswith('#') and line.strip() :
      key=line.strip().split('=')[0]
      exec 'val='+line.strip().split('=')[1]
      CFGVALUES[key]=val

I found other ways to get the values in there but most of the time there would be output like:
Code: Select all
print CFGVALUES
{'files': ["'..mylib/mylib.py'", "'applib.py'", "'app.py'"]}

Does anyone have any suggestions or is my solution the best and/or only way?
Last edited by pnelsonsr on Tue Jan 28, 2014 7:33 pm, edited 1 time in total.
pnelsonsr
 
Posts: 25
Joined: Thu Dec 12, 2013 12:48 am

Re: Best way to bring a list in from a config file

Postby hansn » Tue Jan 28, 2014 12:26 pm

Just a word of advice: edit: Take the advice from the posts down below.

I found other ways to get the values in there but most of the time there would be output like:
[code
]print CFGVALUES
{'files': ["'..mylib/mylib.py'", "'applib.py'", "'app.py'"]}
[/code]

What's wrong with it? Use str.strip("'") to remove the extra single quotes. (That's str.strip(" ' ") with a single quote in the middle)
Last edited by hansn on Wed Jan 29, 2014 8:10 am, edited 2 times in total.
hansn
 
Posts: 87
Joined: Thu Feb 21, 2013 8:46 pm

Re: Best way to bring a list in from a config file

Postby stranac » Tue Jan 28, 2014 12:42 pm

There is a library for parsing .cfg files, you shouldn't be doing it manually...
http://docs.python.org/2/library/configparser.html

It won't give you a real python list, but you can store json lists in the config file, and just use json.loads() to get a python list.
Alternatively, you could just use json for config.
Friendship is magic!

R.I.P. Tracy M. You will be missed.
User avatar
stranac
 
Posts: 1093
Joined: Thu Feb 07, 2013 3:42 pm

Re: Best way to bring a list in from a config file

Postby Kebap » Tue Jan 28, 2014 1:28 pm

I would advise against using exec in that way, or in most any way really.

You never know what the files may contain, then execute arbitrary code, does not seem very wise.
Learn: How To Ask Questions The Smart Way
Join the #python-forum IRC channel on irc.freenode.net and chat with uns directly!
Kebap
 
Posts: 365
Joined: Thu Apr 04, 2013 1:17 pm
Location: Germany, Europe

Re: Best way to bring a list in from a config file

Postby pnelsonsr » Tue Jan 28, 2014 4:42 pm

Kebap wrote:I would advise against using exec in that way, or in most any way really.

You never know what the files may contain, then execute arbitrary code, does not seem very wise.

I never use exec but it was the only way (at the time...) I could get it working and really the ultimate reason for creating this post!

stranac wrote:There is a library for parsing .cfg files, you shouldn't be doing it manually...
http://docs.python.org/2/library/configparser.html

It won't give you a real python list, but you can store json lists in the config file, and just use json.loads() to get a python list.
Alternatively, you could just use json for config.

Perfect a lib to do config parsing-> I'll check it out! Hmm json... This might be a direction that I need to take that I have been avoiding. Every time I start down the json path it seems like a tital wave of info and I just stop... Got any good tutorials to get one going?

hansn wrote:Just a word of advice: don't worry about the 'right' way, just worry about making it work the way you want it to.

What's wrong with it? Use str.strip("'") to remove the extra single quotes. (That's str.strip(" ' ") with a single quote in the middle)

Well that's probably true! Sometimes when I wrestle to get a solution it feels like there just has got to be a better way. This one definitely left me feeling kind of dirty. Hmm... I think I was so focused on this that I was in a mental block. I'll try it with the 'strip("'")' thing!
pnelsonsr
 
Posts: 25
Joined: Thu Dec 12, 2013 12:48 am

Re: Best way to bring a list in from a config file

Postby micseydel » Tue Jan 28, 2014 6:45 pm

JSON is nice because you can often dump whatever Python object you want to a file, and get it back very easily as a Python object (dictionary, list, string). I use JSON often and love it. I feel like it's straightforward.
Code: Select all
import json

my_record = {"key": ["contents"]}

with open("my_file", "w") as my_file:
    json.dump(my_record, my_file)

with open("my_file") as saved_file:
    saved_record = json.load(saved_file)

print saved_record

(Untested, but should be very, very close.)

Perhaps you would prefer config files, since while JSON is considered "human readable" it looks like the config files have less noise and are probably easier for regular users. You can save JSON with nice indentation and such instead of the regular blob form by checking out the option arguments to dump() (or dumps()). Anyway, I hope you can make an informed choice now! Feel free to followup with any questions.
Join the #python-forum IRC channel on irc.freenode.net!
User avatar
micseydel
 
Posts: 1116
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Best way to bring a list in from a config file

Postby pnelsonsr » Tue Jan 28, 2014 7:32 pm

hansn wrote:What's wrong with it? Use str.strip("'") to remove the extra single quotes. (That's str.strip(" ' ") with a single quote in the middle)

I tried the .strip("'") and cool it got rid of the ' prior to dumping it in as a list. But then I got thinking... Oh No not again! With list comprehension in mind, this is what I came up with:
Code: Select all
with open(CFGFILE,'r') as f :
  for line in f :
    if not line.startswith('#') and line.strip() :
      ld=line.strip().split('=')
      CFGVALUES[ld[0]]=[x.strip('\'[]') for x in ld[1].split(',')]

The .strip removes all ' [ and ] prior to putting each item in the list.
This works fine and I don't feel dirty about having to use eval plus it reads well.
pnelsonsr
 
Posts: 25
Joined: Thu Dec 12, 2013 12:48 am

Re: Best way to bring a list in from a config file

Postby ochichinyezaboombwa » Tue Jan 28, 2014 9:41 pm

pnelsonsr wrote:...works fine but what I would really like to do is use a list in the config file like:
Code: Select all
files=['..mylib/mylib.py', 'applib.py', 'app.py']


Beats me: why do you want it like this? Config file is a file that can be easily changed in a text editor. In most cases, you should assume that a user will edit it and create some mess. And it's much easier to mess up what you like (?) than the following:
Code: Select all
files = ..mylib/mylib.py, applib.py, app.py


That said: if you must use Python representation, json is perfect. Look at the example above one more time: how hard is that?
ochichinyezaboombwa
 
Posts: 200
Joined: Tue Jun 04, 2013 7:53 pm

Re: Best way to bring a list in from a config file

Postby pnelsonsr » Wed Jan 29, 2014 12:20 am

micseydel wrote:Perhaps you would prefer config files, since while JSON is considered "human readable" it looks like the config files have less noise and are probably easier for regular users. You can save JSON with nice indentation and such instead of the regular blob form by checking out the option arguments to dump() (or dumps()). Anyway, I hope you can make an informed choice now! Feel free to followup with any questions.

I think I'd prefer a config file. After messing around with json for a bit (dumping and loading) I realized the output isn't what I'd want in a config file. However, I did see how I can use it for state saving and things like that which I'll definitely do.

stranac wrote:There is a library for parsing .cfg files, you shouldn't be doing it manually...
http://docs.python.org/2/library/configparser.html


I messed around a bunch with ConfigParser and agree this is the way to go. It doesn't recognize lists but it allows them as a string (both reading and writing). I'm probably silly for wanting lists but I love lists and dictionaries... What can I say? The main reason for wanting a list here is that ease in using an item (which is a directory) that may or may not have white space in it and so as to not have the config file editor having to do escaping.
pnelsonsr
 
Posts: 25
Joined: Thu Dec 12, 2013 12:48 am

Re: Best way to bring a list in from a config file

Postby pnelsonsr » Wed Jan 29, 2014 8:26 pm

So I went with ConfigParser and this changed things a bit. Ultimately the point here was to make the config file fail nicely with anything the user threw at it. So I also wanted to be able to handle an incoming list from the config file. ConfigParser doesn't parse list values, so I have to grab and deal with that myself. I do this with something like:

with a config file like:
Code: Select all
[Main]
files=['..mylib/mylib.py', 'applib.py', 'app.py']


the code to pull it in as a list:
Code: Select all
files=[x.strip('\'') for x in config.get('Main',var).strip('[]').split(',') if len(x)>0]


This list comprehension is a bit different from above as I found that it didn't play with with a config file option of 'files=[]'. But now it does play well with it or any other blank files option value.
pnelsonsr
 
Posts: 25
Joined: Thu Dec 12, 2013 12:48 am

Re: Best way to bring a list in from a config file

Postby micseydel » Wed Jan 29, 2014 9:32 pm

I doubt strip("[]") is doing as you expect. Also, you probably want to split on ", " rather than "," (note the space) if your list is as you expect, since when you try to strip off the quote character it won't work if there's a space on the end. Also note that if you're creating the file by using str() on the list, that a string inside the list will have a double quote (") if there is a single quote in that string so you won't be stripping the right thing. And if you read that string, then write it again, eventually you can end up with insanity like this
Code: Select all
'\'"That\'s funny"\''

What you're doing is called "rolling your own" and I recommend against it. It won't be robust. You can achieve something much like what you want by making use of the csv module. You don't have to save your whole config file as a csv, you can simply use the module as a robust way of creating the list as a string and retrieving it, but still store that string in your config file.
Join the #python-forum IRC channel on irc.freenode.net!
User avatar
micseydel
 
Posts: 1116
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Best way to bring a list in from a config file

Postby stranac » Wed Jan 29, 2014 9:51 pm

Also, if you just want to create a list out of a string that looks like one, you can just use json.loads() or ast.literal_eval(), depending on what exactly you want.
Friendship is magic!

R.I.P. Tracy M. You will be missed.
User avatar
stranac
 
Posts: 1093
Joined: Thu Feb 07, 2013 3:42 pm

Re: Best way to bring a list in from a config file

Postby micseydel » Wed Jan 29, 2014 10:06 pm

Good point stranac, it hadn't occurred to me that he could use JSON for regular lists! That's a much better solution. I also didn't know about the ast module, which is awesome.
Join the #python-forum IRC channel on irc.freenode.net!
User avatar
micseydel
 
Posts: 1116
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Best way to bring a list in from a config file

Postby pnelsonsr » Fri Jan 31, 2014 3:45 pm

micseydel wrote:I doubt strip("[]") is doing as you expect. Also, you probably want to split on ", " rather than "," (note the space) if your list is as you expect, since when you try to strip off the quote character it won't work if there's a space on the end. Also note that if you're creating the file by using str() on the list, that a string inside the list will have a double quote (") if there is a single quote in that string so you won't be stripping the right thing. And if you read that string, then write it again, eventually you can end up with insanity like this
Code: Select all
'\'"That\'s funny"\''

What you're doing is called "rolling your own" and I recommend against it. It won't be robust. You can achieve something much like what you want by making use of the csv module. You don't have to save your whole config file as a csv, you can simply use the module as a robust way of creating the list as a string and retrieving it, but still store that string in your config file.

Yes I did find that out and I was checking back in to update things. I decided to remove the (or any) space after the comma with (basically this...):
Code: Select all
valstr=config.get('Main','files')
while r.search(', ',valstr) : valstr=valstr.replace(', ',',')
valstr=valstr.strip('[]')
files=[x.strip('\'') for x in valstr.split(',') if len(x)>0]

RYO Yeah thats funny! I checked out the csv module and this seems like a bit over kill for what I'm trying to do. I do like that the module exists though and I may have a need for it shortly to do exactly the csv export import kind of thing! Thanks
Last edited by pnelsonsr on Fri Jan 31, 2014 4:07 pm, edited 1 time in total.
pnelsonsr
 
Posts: 25
Joined: Thu Dec 12, 2013 12:48 am

Re: Best way to bring a list in from a config file

Postby pnelsonsr » Fri Jan 31, 2014 4:00 pm

stranac wrote:Also, if you just want to create a list out of a string that looks like one, you can just use json.loads() or ast.literal_eval(), depending on what exactly you want.

I had tried working with json loads but I was not able to get it working. This is basically what I was trying:
Code: Select all
myvar="['..mylib/mylib.py', 'applib.py', 'app.py']"
mylist=json.loads(myvar)
which errors with "No JSON object could be decoded". After trying many times I gave up on the json loads thing. What was (am) I doing wrong?

As far as ast.literal_eval here is what I tried:
Code: Select all
myvar="['..mylib/mylib.py', 'applib.py', 'app.py']"
mylist=ast.literal_eval(myvar)

And this does produce a list from a list looking string. Nice.
pnelsonsr
 
Posts: 25
Joined: Thu Dec 12, 2013 12:48 am

Re: Best way to bring a list in from a config file

Postby stranac » Fri Jan 31, 2014 4:08 pm

pnelsonsr wrote:What was (am) I doing wrong?

json only uses double quotes:
Code: Select all
>>> json.loads("['..mylib/mylib.py', 'applib.py', 'app.py']")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\json\__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "C:\Python27\lib\json\decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python27\lib\json\decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
>>>
>>> json.loads('["..mylib/mylib.py", "applib.py", "app.py"]')
[u'..mylib/mylib.py', u'applib.py', u'app.py']
Friendship is magic!

R.I.P. Tracy M. You will be missed.
User avatar
stranac
 
Posts: 1093
Joined: Thu Feb 07, 2013 3:42 pm

Re: Best way to bring a list in from a config file

Postby pnelsonsr » Fri Jan 31, 2014 6:37 pm

stranac wrote:json only uses double quotes:
Code: Select all
>>> json.loads('["..mylib/mylib.py", "applib.py", "app.py"]')
[u'..mylib/mylib.py', u'applib.py', u'app.py']

Ahh... So yes this works:
Code: Select all
myvar="['..mylib/mylib.py', 'applib.py', 'app.py']"
mylist=json.loads(myvar.replace('\'','"'))
OK so thats cool, but whats the 'u' in front of each item?

Also, I've been messing around with ast.literal_eval() and this really works well with all types. For example:
Code: Select all
>>> mstr="'this is a string'"
>>> mint='1'
>>> mbol='True'
>>> mnone='None'
>>> mlist="['1','2']"
>>> mtup="('1','2')"
>>> mdic="{'1':'one'}"
>>> for var in ['mstr','mint','mbol','mnone','mlist','mtup','mdic'] :
...   mvar=vars()[var];mval=a.literal_eval(mvar);print('{:<19} -> {:<17} -> {}'.format(mvar,mval,type(mval)))
'this is a string'  -> this is a string  -> <type 'str'>
1                   -> 1                 -> <type 'int'>
True                -> 1                 -> <type 'bool'>
None                -> None              -> <type 'NoneType'>
['1','2']           -> ['1', '2']        -> <type 'list'>
('1','2')           -> ('1', '2')        -> <type 'tuple'>
{'1':'one'}         -> {'1': 'one'}      -> <type 'dict'>
I plugged this into my app and it just works. Just what I needed! So thanks!
pnelsonsr
 
Posts: 25
Joined: Thu Dec 12, 2013 12:48 am

Re: Best way to bring a list in from a config file

Postby stranac » Fri Jan 31, 2014 7:50 pm

pnelsonsr wrote:Ahh... So yes this works:
Code: Select all
myvar="['..mylib/mylib.py', 'applib.py', 'app.py']"
mylist=json.loads(myvar.replace('\'','"'))

Why not just have double quotes in the config file?

pnelsonsr wrote:OK so thats cool, but whats the 'u' in front of each item?

Just means they're unicode strings.

pnelsonsr wrote:Also, I've been messing around with ast.literal_eval() and this really works well with all types.

Well, not all types, just those 7.
But yeah, it can be really useful sometimes.
Friendship is magic!

R.I.P. Tracy M. You will be missed.
User avatar
stranac
 
Posts: 1093
Joined: Thu Feb 07, 2013 3:42 pm

Re: Best way to bring a list in from a config file

Postby pnelsonsr » Fri Jan 31, 2014 8:03 pm

stranac wrote:Why not just have double quotes in the config file?
Just means they're unicode strings.
Well, not all types, just those 7.
But yeah, it can be really useful sometimes.

Well that's true except I don't have total control over users...
Ah, yes...
pnelsonsr
 
Posts: 25
Joined: Thu Dec 12, 2013 12:48 am

Re: Best way to bring a list in from a config file

Postby micseydel » Sat Feb 01, 2014 3:16 am

I never write JSON by hand. I create a Python structure and use json.dumps(). You can complain to a user if they violate the syntax, or even place a comment in the config file letting them know.

It sounds like ast is best for your situation though. I didn't know about it before this thread, very cool.
Join the #python-forum IRC channel on irc.freenode.net!
User avatar
micseydel
 
Posts: 1116
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Next

Return to General Coding Help

Who is online

Users browsing this forum: No registered users and 3 guests