Re: [Xen-devel] Hypervisor build can't work without `python` (Was: Re: Xen commit 9b0bc91b3 possibly removed too much info from README)

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 17/04/2019 12:43, Juergen Gross wrote:
> On 17/04/2019 13:26, Wei Liu wrote:
>> On Wed, Apr 17, 2019 at 12:23:29PM +0100, Andrew Cooper wrote:
>>> On 17/04/2019 12:16, Wei Liu wrote:
>>>> On Wed, Apr 17, 2019 at 12:15:04PM +0100, Andrew Cooper wrote:
>>>>> On 17/04/2019 12:03, Wei Liu wrote:
>>>>>> On Wed, Apr 17, 2019 at 11:58:49AM +0100, Andrew Cooper wrote:
>>>>>>> On 17/04/2019 11:57, Wei Liu wrote:
>>>>>>>> On Wed, Apr 17, 2019 at 11:44:36AM +0100, Andrew Cooper wrote:
>>>>>>>>> On 17/04/2019 11:41, Wei Liu wrote:
>>>>>>>>>> On Wed, Apr 17, 2019 at 10:56:57AM +0100, Wei Liu wrote:
>>>>>>>>>>>> Here's what i did having pulled the master at commit cb70a26
>>>>>>>>>>>>
>>>>>>>>>>>> tar xf /path/to/xen-cb70a26.tar.gz
>>>>>>>>>>>>
>>>>>>>>>>>> cd xen-master/
>>>>>>>>>>>>
>>>>>>>>>>>> PYTHON=/usr/bin/python3 ./configure --prefix=/usr   \
>>>>>>>>>>>>    --disable-seabios   \
>>>>>>>>>>>>    --disable-qemu-traditional  \
>>>>>>>>>>>>    --disable-rombios     \
>>>>>>>>>>>>    --disable-stubdom 2>&1 | tee ../config.log
>>>>>>>>>>>>
>>>>>>>>>>>> make EFI_DIR=/usr/lib/efi world 2>&1 | tee ../make_world_3.out
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ...
>>>>>>>>>>>> gcc -Wp,-MD,tools/kconfig/.zconf.tab.o.d    -D_GNU_SOURCE
>>>>>>>>>>>> -D_DEFAULT_SOURCE -DCURSES_LOC="<ncurses.h>" -DNCURSES_WIDECHAR=1
>>>>>>>>>>>> -DLOCALE  -Itools/kconfig -c -o tools/kconfig/zconf.tab.o
>>>>>>>>>>>> tools/kconfig/zconf.tab.c
>>>>>>>>>>>> gcc  -o tools/kconfig/conf tools/kconfig/conf.o 
>>>>>>>>>>>> tools/kconfig/zconf.tab.o
>>>>>>>>>>>> tools/kconfig/conf -s --silentoldconfig Kconfig
>>>>>>>>>>>> make[3]: Leaving directory '/usr/src/xen/xen-master/xen'
>>>>>>>>>>>> make -f Rules.mk _install
>>>>>>>>>>>> make[3]: Entering directory '/usr/src/xen/xen-master/xen'
>>>>>>>>>>>> make -C tools
>>>>>>>>>>>> make[4]: Entering directory '/usr/src/xen/xen-master/xen/tools'
>>>>>>>>>>>> make symbols
>>>>>>>>>>>> make[5]: Entering directory '/usr/src/xen/xen-master/xen/tools'
>>>>>>>>>>>> gcc -Wall -Werror -Wstrict-prototypes -O2 -fomit-frame-pointer
>>>>>>>>>>>> -fno-strict-aliasing -Wdeclaration-after-statement -o symbols
>>>>>>>>>>>> symbols.c
>>>>>>>>>>>> make[5]: Leaving directory '/usr/src/xen/xen-master/xen/tools'
>>>>>>>>>>>> make[4]: Leaving directory '/usr/src/xen/xen-master/xen/tools'
>>>>>>>>>>>> make -f /usr/src/xen/xen-master/xen/Rules.mk include/xen/compile.h
>>>>>>>>>>>> make[4]: Entering directory '/usr/src/xen/xen-master/xen'
>>>>>>>>>>>>  Xen 4.13-unstable
>>>>>>>>>>>> /bin/sh: python: command not found
>>>>>>>>>>>> make[4]: *** [Makefile:169: include/xen/compile.h] Error 127
>>>>>>>>>>> Here. The PYTHON invocation is from the hypervisor build, not the 
>>>>>>>>>>> tools
>>>>>>>>>>> build.
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the log.  I will have a closer look.
>>>>>>>>>>>
>>>>>>>>>>>> make[4]: Leaving directory '/usr/src/xen/xen-master/xen'
>>>>>>>>>>>> make[3]: *** [Makefile:137: /usr/src/xen/xen-master/xen/xen] Error 
>>>>>>>>>>>> 2
>>>>>>>>>>>> make[3]: Leaving directory '/usr/src/xen/xen-master/xen'
>>>>>>>>>>>> make[2]: *** [Makefile:45: install] Error 2
>>>>>>>>>>>> make[2]: Leaving directory '/usr/src/xen/xen-master/xen'
>>>>>>>>>>>> make[1]: *** [Makefile:123: install-xen] Error 2
>>>>>>>>>>>> make[1]: Leaving directory '/usr/src/xen/xen-master'
>>>>>>>>>>>> make: *** [Makefile:165: world] Error 2
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Note the
>>>>>>>>>>>>
>>>>>>>>>>>> /bin/sh: python: command not found
>>>>>>>>>>>>
>>>>>>>>>> The issue here is on LFS there is no /usr/bin/python. So the 
>>>>>>>>>> hypervisor
>>>>>>>>>> build is broken because xen/include/compile.h invokes
>>>>>>>>>>
>>>>>>>>>>  @$(PYTHON) tools/fig-to-oct.py < .banner >> $@.new
>>>>>>>>>>
>>>>>>>>>> There are other appearances of PYTHON in hypervisor build, which are
>>>>>>>>>> equally broken.
>>>>>>>>>>
>>>>>>>>>> We had / have a policy to avoid making hypervisor build dependent on
>>>>>>>>>> configure. I have CC'ed more people on their opinions.
>>>>>>>>> Where has $(PYTHON) got lost?  It was specified correctly to begin 
>>>>>>>>> with.
>>>>>>>> Where is it specified?
>>>>>>> Oh - I see now.
>>>>>>>
>>>>>>> It is specified for ./configure, but not for make.
>>>>>> Right.
>>>>>>
>>>>>> It works for tools because tools build includes config/Tools.mk which
>>>>>> contains the correct python path. Tools.mk is generated by configure.
>>>>>>  
>>>>>> Obviously hypervisor build shouldn't include Tools.mk. Furthermore, the
>>>>>> policy of "hv shouldn't depend on configure" makes this route a no-go.
>>>>>>
>>>>>> Either we change the policy, or we document the deficiency, or we come
>>>>>> up with some clever trick to probe.
>>>>> Given the shebang in all of our scripts, would `env python` be
>>>>> sufficiently clever trickery, or is that a can of worms we'd prefer not
>>>>> to open?
>>>> I already tried env python -- it didn't work. I think it searches PATHS
>>>> just like a shell would.
>>> So this particular use of python is completely trivial.  It is an octal
>>> conversion on the banner string, to feed into a string for XEN_BANNER
>>>
>>> It exists because I removed a fork of figlet[1] from Xen's source tree
>>> with this script, but its probably just as easy to do with awk or
>>> something similar.
>>>
>>> ~Andrew
>>>
>>> [1] Yes - Xen did use to carry a fork of figlet which differed solely by
>>> being able to output an octal-escaped string.  No - I have no words
>>> which can possibly be used to defend this...
>>
>> But there are other less trivial ones. See xen/include/Makefile. XSM
>> also relies on it to generate policy. What do you want to do with those?
> Shouldn't _some_ settings of configure end up in the hypervisor build,
> too? Like CC, AWK, SHELL, HOSTCC?
>
> Maybe we need a rather small basic_tools.mk containing those settings
> (with a sane default, of course)?

In the past, there has been a vehement objection to requiring
./configure before building Xen, and I still agree with this point of
view.  Noone should need to run ./configure to be able to build a
freestanding item with no userspace connection.

The problem Xen has is that we've got two vastly different projects in
the same source repo, and `make` runes which encompass both.

Could we go with some compromise where the results of configure are used
if they are available, but for people not wanting to run configure, if
they've got non-standard standard tools, they need to pass them to make
as well?  This is consistent with how cross compiling works.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel