Parse array of strings, integers,dict using Python C/C++ API

This is the place for queries that don't fit in any of the other categories.

Parse array of strings, integers,dict using Python C/C++ API

Postby lovecodecakes » Tue Aug 13, 2013 7:48 pm

This is refering to my earlier question: Python C API: Parse args of string and integer in C >>(@stackoverflow: http://stackoverflow.com/questions/1804 ... teger-in-c )

and then I referred to

http://code.activestate.com/lists/python-list/31841/ for some clues.

i was able to parse a list
Code: Select all
if (!PyArg_ParseTuple(args, "O!", &PyList_Type, &listObj))

how can i similarly parse a bunch of lists?

something like this i highly doubt would work:

Code: Select all
if (!PyArg_ParseTuple(args, "O!O!O!O!", &PyList_Type, &listObj1,&PyList_Type, &listObj2,&PyList_Type, &listObj3,&PyList_Type, &listObj4)

also is it possible if i convert the list into an array say

Code: Select all
l = [bunch of strings]

and convert it like:

Code: Select all
l = (ctypes.c_char_p* len(l))(*l)

and then parse from C? How can that be achieved?

Similarly, instead of lists and using ctypes, how can I parse an array of integers?

also checked out: PySequence_Fast >> http://effbot.org/zone/python-capi-sequences.htm but I think

Code: Select all
strObj = PyList_GetItem(listObj, i); /* Can't fail */

/* make it a string */
line = PyString_AsString( strObj );

and

Code: Select all
seq = PySequence_Fast(obj, "expected a sequence");
for (i = 0; i < len; i++) {
    item = PySequence_Fast_GET_ITEM(seq, i);

arent they similar?

Ultimately, i want to parse a bunch of arrays of strings and integers from C.

Also, is it possible if I could get a working example of parsing dictionaries from Python to C++ and back?
lovecodecakes
 
Posts: 56
Joined: Mon Feb 11, 2013 8:19 pm

Re: Parse array of strings, integers,dict using Python C/C++

Postby micseydel » Tue Aug 13, 2013 8:49 pm

Just as a heads up, I don't think any regulars here use the C API. casevh probably does (he wrote a couple of tutorials about the C API*), and he might respond but I'm not sure how often he visits or how closely he keeps up with posts.

*http://python-forum.org/viewtopic.php?f=25&t=621
viewtopic.php?f=25&t=3365
Join the #python-forum IRC channel on irc.freenode.net!
User avatar
micseydel
 
Posts: 940
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Parse array of strings, integers,dict using Python C/C++

Postby lovecodecakes » Tue Aug 13, 2013 9:10 pm

micseydel wrote:*http://python-forum.org/viewtopic.php?f=25&t=621
viewtopic.php?f=25&t=3365


Thanks! Checked out his posts. works on c api alot and some nice logic
lovecodecakes
 
Posts: 56
Joined: Mon Feb 11, 2013 8:19 pm

Re: Parse array of strings, integers,dict using Python C/C++

Postby micseydel » Tue Aug 13, 2013 9:16 pm

Did it help you solve your problem posted here?
Join the #python-forum IRC channel on irc.freenode.net!
User avatar
micseydel
 
Posts: 940
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Parse array of strings, integers,dict using Python C/C++

Postby lovecodecakes » Tue Aug 13, 2013 9:23 pm

micseydel wrote:Did it help you solve your problem posted here?

Not at all.
I did go through his Part 2 take on C Python API.
Apart from Undefined function definitions. always good to learn though.

Seems to me not much C api crowd here?
lovecodecakes
 
Posts: 56
Joined: Mon Feb 11, 2013 8:19 pm

Re: Parse array of strings, integers,dict using Python C/C++

Postby casevh » Tue Aug 13, 2013 10:19 pm

I'm still here. I've been quite busy at home and at work.

Note: Code is untested. Code is missing variable declarations.

i was able to parse a list

Code: Select all
if (!PyArg_ParseTuple(args, "O!", &PyList_Type, &listObj))

how can i similarly parse a bunch of lists?

something like this i highly doubt would work:

Code: Select all
if (!PyArg_ParseTuple(args, "O!O!O!O!", &PyList_Type, &listObj1,&PyList_Type, &listObj2,&PyList_Type, &listObj3,&PyList_Type, &listObj4)


I haven't tried it, but I think your second example should work. But it does assume all four parameters are lists. If your function is not exposed directly to an end user but is only called by another Python function that validates the input, then this approach is okay. If not, you might just want to use:

Code: Select all
if (!PyArg_ParseTuple(args, "OOOO", &arg1, &arg2, &arg3, &arg4)


and then process each argument separately.

Assuming arg1 contains a sequence (list. tuple, interable) of integers, the code would look something like this:

Code: Select all
temp = PySequence_Fast(arg1, "argument must be iterable");
if (!temp) {
    return NULL;
}
length1 = PySequence_Fast_GET_SIZE(temp);
array1 = malloc(length1 * sizeof(long);
if (!array1) {
    Py_DECREF(temp);
    return PyErr_NoMemory();
}
for (i=0; i < length1; i++) {
    t = PyLong_AsLong(PySequence_Fast_GET_ITEM(temp, i));
    if (t == -1 && PyErr_Occurred()) {
        Py_DECREF(temp);
        free(array1);
        return NULL;
    }
    array1[i] = temp;
}

# Do magic stuff.
# Assume array1 contents have changed and need to be returned.

result1 = PyTuple_New(length1);
if (!result1) {
    Py_DECREF(temp);
    free(array1);
    return NULL;
}
for (i=0; i < length1; i++) {
    z = PyLong_FromLong(array1[i]);
    if (!z)  {
        Py_DECREF(temp);
        free(array1);
        return NULL;
    }
    PyTuple_SET_ITEM(result1, i, z);
}


If all four of your arguments have the same length, you might want to first convert them into a list of 4-tuples in Python. Then your C function just needs to accept a single argument. Then loop over the list and parse each tuple with PyArg_ParseTuple. The C code would be easier to write but it may be slower due to the list/tuple create in Python.

Does this help?

casevh

BTW, is there a bug in my second C API tutorial?
casevh
 
Posts: 56
Joined: Sat Feb 09, 2013 7:35 am

Re: Parse array of strings, integers,dict using Python C/C++

Postby casevh » Wed Aug 14, 2013 5:00 am

[another comment, and a bump since I edited my previous post....]

Another possibility (that I've never tried) is the array module and accessing an instance via the memoryview interface.

casevh
casevh
 
Posts: 56
Joined: Sat Feb 09, 2013 7:35 am

Re: Parse array of strings, integers,dict using Python C/C++

Postby lovecodecakes » Wed Aug 14, 2013 11:52 am

casevh wrote:
Does this help?

casevh

BTW, is there a bug in my second C API tutorial?


Hi, I just checked this. let me test this code and I will paste here how it fares with the c snippet and the test python code code as well.
Thanks!
I didn't run the examples in your 2nd c api tutorial.I should try that out as well sometime soon.
lovecodecakes
 
Posts: 56
Joined: Mon Feb 11, 2013 8:19 pm

Re: Parse array of strings, integers,dict using Python C/C++

Postby lovecodecakes » Wed Aug 14, 2013 11:53 am

casevh wrote:Another possibility (that I've never tried) is the array module and accessing an instance via the memoryview interface.
casevh

what is memoryview interface?
a c array by converting a list into an array first?(Use of ctypes?)
lovecodecakes
 
Posts: 56
Joined: Mon Feb 11, 2013 8:19 pm

Re: Parse array of strings, integers,dict using Python C/C++

Postby lovecodecakes » Wed Aug 14, 2013 9:35 pm

casevh wrote:I'm still here. I've been quite busy at home and at work.

Does this help?



Try installing the following as python lib. the code below should print few gibberish
as sample.d(["324","2","fsdf"])
i dont know how to retrieve it as an array of char and print it all out.
or return it all out.
or again build a tuple/list of strings and print it out.

Also could you tell me how to parse a dictionary to map in c++ and return it back as dict?

Code: Select all
//API header file - The header file Python.h.
#include </usr/include/python2.7/Python.h>
#include <string>
#include <vector>
//for list parsing see: http://mail.python.org/pipermail/tutor/2005-February/035686.html

using namespace std;
//The C functions
/* Except error handling variable */
static PyObject *error;

static PyObject* sample_d(PyObject *self, PyObject *args)
{
    int i =0;
    char line;
    Py_ssize_t size;
    vector<string> t_array;
    PyObject *tseq, *arg1, *z;
    char * arg2;
    int t_seqlen=0;
    printf("OK \n");
    /*Check if the parsed argument value is a actually parsed or not as a string.
    If not return NULL */
    if (!PyArg_ParseTuple(args, "O", &arg1))
    {
        printf("OK. inside !PyArg....");
        return NULL;
    }
    t = PySequence_Fast(arg1, "argument must be iterable");
    if (!t) {
        return NULL;
    }
    t_seqlen = PySequence_Fast_GET_SIZE(t);

    for (i=0; i < t_seqlen; i++)
    {
        arg2 = PyString_AsString(PySequence_Fast_GET_ITEM(t, i));
        printf("%c \n", arg2[i]);
        if (PyErr_Occurred()) {
            Py_DECREF(t);
            //free(t_array);
            return NULL;
        }
    }

    Py_RETURN_NONE;
}

/*A table mapping the names of Python functions to C functions
inside the extension module
For this we use PyMethoDef which is a C structure
struct PyMethodDef {
   char *ml_name; //Name used by Python interpreter
   PyCFunction ml_meth; // module_func name
   int ml_flags; // METH_VARARGS, METH_NOARGS, METH_KEYWORDS
   char *ml_doc; // docstring for the function, which could be NULL
};
*/
static PyMethodDef py_methods[] = {
    //"Python Name"    C-func Name    arg_presentation    desc
    {"d",            sample_d,    METH_VARARGS,        "some description},
    {NULL, NULL, 0, NULL}    /*Sentinel*/
    };

/*Following module initializes the module post that we have written
- An initialization function */
PyMODINIT_FUNC initpost(void){
    PyObject *m;
    m = Py_InitModule("sample", py_methods); //module name, PyMethodDEf defined above, doc_string
    if (m == NULL) return;
    //create post.error python error object
    error = PyErr_NewException("sample.error", NULL, NULL);
    Py_INCREF(error);
    PyModule_AddObject(m, "error", error);
    }
lovecodecakes
 
Posts: 56
Joined: Mon Feb 11, 2013 8:19 pm

Re: Parse array of strings, integers,dict using Python C/C++

Postby casevh » Thu Aug 15, 2013 3:21 am

lovecodecakes wrote:
Try installing the following as python lib. the code below should print few gibberish
as sample.d(["324","2","fsdf"])
i dont know how to retrieve it as an array of char and print it all out.
or return it all out.
or again build a tuple/list of strings and print it out.

Also could you tell me how to parse a dictionary to map in c++ and return it back as dict?

Code: Select all
//API header file - The header file Python.h.
#include </usr/include/python2.7/Python.h>
#include <string>
#include <vector>
//for list parsing see: http://mail.python.org/pipermail/tutor/2005-February/035686.html

using namespace std;
//The C functions
/* Except error handling variable */
static PyObject *error;

static PyObject* sample_d(PyObject *self, PyObject *args)
{
    int i =0;
    char line;
    Py_ssize_t size;
    vector<string> t_array;
    PyObject *tseq, *arg1, *z;
    char * arg2;
    int t_seqlen=0;
    printf("OK \n");
    /*Check if the parsed argument value is a actually parsed or not as a string.
    If not return NULL */
    if (!PyArg_ParseTuple(args, "O", &arg1))
    {
        printf("OK. inside !PyArg....");
        return NULL;
    }
    t = PySequence_Fast(arg1, "argument must be iterable");
    if (!t) {
        return NULL;
    }
    t_seqlen = PySequence_Fast_GET_SIZE(t);

    for (i=0; i < t_seqlen; i++)
    {
        arg2 = PyString_AsString(PySequence_Fast_GET_ITEM(t, i));
        printf("%c \n", arg2[i]);
        if (PyErr_Occurred()) {
            Py_DECREF(t);
            //free(t_array);
            return NULL;
        }
    }

    Py_RETURN_NONE;
}

/*A table mapping the names of Python functions to C functions
inside the extension module
For this we use PyMethoDef which is a C structure
struct PyMethodDef {
   char *ml_name; //Name used by Python interpreter
   PyCFunction ml_meth; // module_func name
   int ml_flags; // METH_VARARGS, METH_NOARGS, METH_KEYWORDS
   char *ml_doc; // docstring for the function, which could be NULL
};
*/
static PyMethodDef py_methods[] = {
    //"Python Name"    C-func Name    arg_presentation    desc
    {"d",            sample_d,    METH_VARARGS,        "some description},
    {NULL, NULL, 0, NULL}    /*Sentinel*/
    };

/*Following module initializes the module post that we have written
- An initialization function */
PyMODINIT_FUNC initpost(void){
    PyObject *m;
    m = Py_InitModule("sample", py_methods); //module name, PyMethodDEf defined above, doc_string
    if (m == NULL) return;
    //create post.error python error object
    error = PyErr_NewException("sample.error", NULL, NULL);
    Py_INCREF(error);
    PyModule_AddObject(m, "error", error);
    }


It is a standard rule in this forum to post the full error message. For Python code, this usually means including the full text of the exception message. You stated it printed gibberish. Include the text printed to the screen. Format it as a code block so any text alignment isn't lost.

If you want us to run some C code, please make sure it compiles. The code you posted had multiple syntax errors. Also include the setup.py file you used to compile the code.

I fixed the syntax errors in your code and fixed several other errors. There were probably two errors in your printf("%c \n", arg2[i]) statement. The %c format code only prints a single character. From your previous comments, I assume you were trying to print the entire string. arg2[i] probably isn't doing what you expect - you are printing the i-th character of each string. Your gibberish probably started with a "3" (when i is 0), then a NULL character (when i is 1), and then a "d" (when i is 2). If you want to print the first character of each string, you should use printf("%c \n", arg2[0]). If you want to print the entire string, you should use printf("%s \n", arg2).

Here is the setup.py file I used:

Code: Select all
from distutils.core import setup, Extension

case_module = Extension('sample', sources = ['sample.c'])

setup (name = 'sample',
       version = '0.0',
       description = 'sample',
       ext_modules = [case_module])

Here is sample.c:

Code: Select all
#include "Python.h"

/* Except error handling variable */
static PyObject *error;

static PyObject* sample_d(PyObject *self, PyObject *args)
{
    int i = 0;
    PyObject *tseq, *arg1;
    char * arg2;
    int t_seqlen=0;

    /* Check if a single argument was passed. We still don't know if it is
       a list/tuple/iterable. */

    if (!PyArg_ParseTuple(args, "O", &arg1)) return NULL;

    tseq = PySequence_Fast(arg1, "argument must be iterable");
    if (!tseq) return NULL;

    t_seqlen = PySequence_Fast_GET_SIZE(tseq);

    for (i=0; i < t_seqlen; i++)
    {
        arg2 = PyString_AsString(PySequence_Fast_GET_ITEM(tseq, i));
        if (!arg2) {
            PyErr_SetString(PyExc_TypeError, "only strings are supported");
            Py_DECREF(tseq);
            return NULL;
        }
        /* arg2 is a pointer to a null-terminated array of characters. It is
           not an array of pointers. So I don't think you should be trying to
           index it via "i". If you are trying to print just the first
           character of the string, then you should use:

               printf("%c \n", arg2[0])

           If you are trying to print the entire string, then you should use:

               printf("%s \n", arg2)
        */
        printf("%s \n", arg2);
    }

    Py_DECREF(tseq);

    Py_RETURN_NONE;
}

static PyMethodDef py_methods[] = {
    {"d", sample_d, METH_VARARGS, "sample function"},
    {NULL, NULL, 0, NULL}
};

PyMODINIT_FUNC initsample(void) {
    PyObject *m;

    m = Py_InitModule("sample", py_methods);
    if (m == NULL) return;

    error = PyErr_NewException("sample.error", NULL, NULL);
    Py_INCREF(error);
    PyModule_AddObject(m, "error", error);
}

Here is the output:

Code: Select all
>>> import sample
>>> dir(sample)
['__doc__', '__file__', '__name__', '__package__', 'd', 'error']
>>> sample.d(["324", "2", "fsdf"])
324
2
fsdf
>>>


Sorry, but I don't do C++.

If you have more questions, can you explain what is your end goal? Are you trying to interface with an existing C library? Are trying to speed up some Python code?

casevh
casevh
 
Posts: 56
Joined: Sat Feb 09, 2013 7:35 am

Re: Parse array of strings, integers,dict using Python C/C++

Postby lovecodecakes » Thu Aug 15, 2013 9:29 am

casevh wrote:
Sorry, but I don't do C++.

If you have more questions, can you explain what is your end goal? Are you trying to interface with an existing C library? Are trying to speed up some Python code?

casevh


First of all, thanks for pointing out that i was using an iterator i to print char value over arg2.(the gibberish)

Im also not used to c++ and its STL.
im trying to , as my end, rather than take in inputs as arrays of char, input it as map which c++ allows.
it will be processed and returned back to python. yes, speeding it up.because there are a few computations happening over the data.
I suppose you dont want me to still include the setup.py and error because you have pointed out the errors correctly and included setup yourself.
lovecodecakes
 
Posts: 56
Joined: Mon Feb 11, 2013 8:19 pm

Re: Parse array of strings, integers,dict using Python C/C++

Postby lovecodecakes » Thu Aug 15, 2013 3:55 pm

casevh wrote:
casevh


Here is the final code

Code: Select all
//API header file - The header file Python.h.
#include </usr/include/python2.7/Python.h>
#include <string>
#include <iostream>
#include <map>

using namespace std;
//The C functions
/* Except error handling variable */
static PyObject *error;

typedef map<string, int> StringIntMap;
typedef map<string, int>::iterator StringIntMapIter;

int DT(StringIntMap inMap, StringIntMap &outList)
{
some foo()
}

static PyObject* post_dynamic(PyObject *self, PyObject *args)
{
    //Declarations here. Not including in this code snippet
    /*Check if the parsed argument value is a actually parsed or not as a string.
    If not return NULL */
    if (!PyArg_ParseTuple(args, "O", &arg1)
    //if (!PyArg_ParseTuple(args, "O", &dct))
    {
        printf("OK. inside !PyArg....");
        return NULL;
    }
    //return Py_BuildValue("O", dct);
    t = PySequence_Fast(arg1, "argument must be iterable");
    if (!t) {
        return NULL;
    }

    t_seqlen = PySequence_Fast_GET_SIZE(tags);

    StringIntMap tMap;
    StringIntMap tOutMap;
    int temp=0;
    /*temporary arrangment to make maps*/
    for (i=0; i < t_seqlen; i++)
    {
        arg2 = PyString_AsString(PySequence_Fast_GET_ITEM(t, i));
        i++;
        temp = PyInt_AsLong(PySequence_Fast_GET_ITEM(t, i));
        tMap[string(arg2)] = temp;

        if (PyErr_Occurred()) {
            Py_DECREF(t);
            return NULL;
        }
        printf("\n %s \n", arg2);
    }

    DT(tMap, tOutMap.); //does something on tOutMap.
    char **outTA = (char **)malloc(tOutMap.size() * sizeof(char *));
    int *outTV = (int *)malloc(tOutMap.size() * sizeof(int));
    i = 0;
   char *oneT = NULL;
   for (StringIntMapIter iter = tOutMap.begin(); iter != tOutMap.end(); ++iter, i++)
   {
      cout << (*iter).first << endl;
      oneT=new char[((iter->first).size() * sizeof(char))+1];
      oneT[(iter->first).size()]=0;
      memcpy(oneT,(iter->first).c_str(),(iter->first).size());
      outTA[i] = oneT;
      outTV[i] = (*iter).second;
   }

    free(outTA);
    free(outTV);
    Py_DECREF(t);
    //Py_DECREF(arg1);
    Py_RETURN_NONE;
}


The question is how to create a Tuple of outTA & outTV and return it back as 2 tuples using Py_BuildValue if the number of elements is not known inside?

something like return Py_BuildValue("()()",outTA, outTV)

outTA is all array of strings and outTV arr of ints.
lovecodecakes
 
Posts: 56
Joined: Mon Feb 11, 2013 8:19 pm

Re: Parse array of strings, integers,dict using Python C/C++

Postby casevh » Fri Aug 16, 2013 3:10 am

lovecodecakes wrote:The question is how to create a Tuple of outTA & outTV and return it back as 2 tuples using Py_BuildValue if the number of elements is not known inside?

something like return Py_BuildValue("()()",outTA, outTV)

outTA is all array of strings and outTV arr of ints.


You can't use Py_BuildValue() for dynamically sized objects. Create tuples of the appropriate length, and then put values into the tuple. Then return the two tuples.

Untested, incomplete code:

Code: Select all
result1 = PyTuple_New(t_seqlen);
result2 = PyTuple_New(t_seqlen);
if (!result1 || !result2) {
   /* error occurred, free memory and reference counts */
}
for (i=0; i < t_seqlen; i++) {
    /* make string object */
    str1 = PyString_FromString();
    if (!str1) {
        /* cleanup */
    }
    int1 = PyInt_FromLong(...);
    if (!int1) {
        /* cleanup */
    }
    PyTuple_SET_ITEM(result1, i, str1);
    PyTuple_SET_ITEM(result2, i, int1);
}

/* cleanup everything except result1 and result2 */

return Py_BuildValue("NN", result1, result2);
casevh
 
Posts: 56
Joined: Sat Feb 09, 2013 7:35 am


Return to General Coding Help

Who is online

Users browsing this forum: No registered users and 1 guest