Xen project Mailing List

Hi Jesus,

While using the Elasticsearch python library

(https://elasticsearch-py.readthedocs.io/en/master/) to add mbox

messages to an index, I would get a UnicodeEncodeError:

"'utf-8' codec can't encode character '\udca0' in position 767:

surrogates not allowed".

Investigating in Grimoire elk https://github.com/grim

oirelab/GrimoireELK/blob/96b00bc682485976104a6825ca63ae0

8639deacc/grimoire_elk/elk/mbox.py#L200 seems to show that

perhaps that tool instead uses Latin-1 encoding, but I found that

to then produce a serialization error (their custom error message:

"Unable to serialize %r (type: %s)"). I suppose this is because

now it's bytes; of course, converting back to string after encoding

just cycles back to the first error.

As somewhat of a Python newbie I don't really know how to tackle

this! My thought atm is to splice the offending character out

of the message.

And to clarify, my understanding is that the final result of this task

is an index of Xen data, with two types: commits and messages.

Each commit document should contain its original information

from git, plus the name of the branch it was developed in. And

should only the mbox messages which appear to be associated

with a specific commit exist in the final index? Is there some

key information in messages that is supposed to indicate the

association of a given commit with a git branch? I would be

grateful if you could specify the end goal a little more. :D

Thanks so much!

Heather

On Sat, Apr 8, 2017 at 10:02 AM, Jesus M. Gonzalez-Barahona <jgb@xxxxxxxxxxxx> wrote:

On Fri, 2017-04-07 at 15:49 -0700, Heather Booker wrote:
> Hi Jesus,
>
> Thanks for your reply!
>
> So about the task, instructions say after analyzing mboxes with
> Perceval to
> "store the resulting raw index in ElasticSearch" - what does raw
> index mean?

In this context, I mean "storing the JSON documents produced by
Perceval in an ElasticSearch index, as such". ElasticSearch stores JSON
documents, so it is just uploading the output of Perceval to it.

> In terms of figuring out the elasticsearch structure, do I want an
> index
> (xen-devel mbox) with a type (message) and each object from the
> perceval
> output to be one document? Or should it be more fine-grained?

Exactly.

Saludos,

Jesus.

> Cheers,
>
> Heather
>
> On Thu, Apr 6, 2017 at 7:05 AM, Jesus M. Gonzalez-Barahona <jgb@biter
> gia.com> wrote:
> > On Wed, 2017-04-05 at 16:43 -0700, Heather Booker wrote:
> > > Hi!
> > >
> > > I'd love to work on the Code Review Dashboard project for this
> > round
> > > of Outreachy.
> >
> > Great!!
> >
> > > Are the steps outlined
> > > here http://markmail.org/message/7adkmords3imkswd still the first
> > > contribution you'd like to see?
> >
> > Yes.
> >
> > > So is this a project that has been worked on in previous rounds
> > of
> > > GSOC/Outreachy also?
> > > If so is there a place to find links to the previous participants
> > > blogs? :)
> >
> > No. We had one participation at some point, but couldn't even start
> > for
> > personal reasons. There are some people considering working on this
> > for
> > this next round of Outreachy, however. You'll see their messages in
> > this mailing list.
> >
> > > Should questions about how the specifications/completion of the
> > > microtask be addressed to
> > > IRC or this list? If IRC, which channel - #xen-opw or #metrics-
> > > grimoire? On that note, I'm
> > > curious why #metrics-grimoire is the listed channel on the
> > project
> > > page - are main contributors
> > > involved in both projects? Or is it just because the Xen
> > dashboard
> > > doesn't have a channel?
> >
> > The code review is for the Xen project, but it is done with (I
> > mean,
> > the ssoftware used for it is) GrimoireLab, which for historical
> > reasons
> > uses the #metrics-grimoire channel. That's why it is likely that
> > you
> > find somebody from the project there.
> >
> > If you have questions, and find me around in IRC, please ping me.
> > If
> > I'm not available, please send an email message.
> >
> > Saludos,
> >
> > Jesus.
> >
> > > Thanks!
> > >
> > > Heather
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@xxxxxxxxxxxxx
> > > https://lists.xen.org/xen-devel
> > --
> > Bitergia: http://bitergia.com
> > /me at Twitter: https://twitter.com/jgbarah
> >
> >
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> https://lists.xen.org/xen-devel
--
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard