to Erlang

Why do we need modules at all?

This is a brain-dump-stream-of-consciousness-thing. I’ve been
thinking about this for a while.

I’m proposing a slightly different way of programming here
The basic idea is

    – do away with modules
    – all functions have unique distinct names
    – all functions have (lots of) meta data
    – all functions go into a global (searchable) Key-value database
    – we need letrec
    – contribution to open source can be as simple as
      contributing a single function
    – there are no “open source projects” – only “the open source
      Key-Value database of all functions”
    – Content is peer reviewed

These are discussed in no particular order below:

Why does Erlang have modules?

There’s a good an bad side to modules:

Good: Provides a unit of compilation, a unit of code
distribution. unit of code replacement

Bad: It’s very difficult to decide which module to put an individual
function in. Break encapsulation (see later)

Aside: lib_misc.erl

When I’m programming I often get to the point were I say there should
a function foo/2 in lists.erl but their isn’t. There should be but
there isn’t – foo/2 is a small self contained thing. Why should it be
in lists.erl because it “feels right”.

Strings are lists, so why do we have two modules lists.erl and
string.erl how should I decide in which module my new string/list
processing function should go.

To avoid all mental anguish when I need a small function that
should be somewhere else and isn’t I stick it in
a module elib1_misc.erl.

My elib1_misc exports the following:

added_files/2                 make_challenge/0
as_bits/1                     make_response/2
as_bits_test/0                make_response_test/0
bdump/2                       make_test_strings/1
bin2hex/1                     make_test_strings_test/0
bin2hex_test/0                make_tmp_filename/2
check_io_list/1               merge_kv/1
collect_atom/1                merge_kv_test/0
collect_atom_test/0           mini_shell/0
collect_int/1                 module_info/0
collect_int_test/0            module_info/1
collect_string/1              ndots/1
collect_string_test/0         nibble_to_hex_char/1
collect_word/1                nibble_to_hex_char_test/0
complete/2                    odd/1
complete_test/0               on_exit/2
dos2unix/1                    out_of_date/2
downcase_char/1               outfile/2
dump/2                        padd/2
dump_tmp/2                    perms/1
duplicates/1                  perms_test/0
ensure_started/2              pmap/2
eval_file/1                   pmap1/2
eval_file_test/0              pmap1_test/0
eval_string/1                 pmap_test/0
eval_string_test/0            priority_receive/0
every/3                       random_seed/0
expand_env_vars/1             random_string/1
expand_file_template/3        random_string/2
expand_string_template/2      read_at_most_n_lines/2
expand_tabs/1                 read_at_most_n_lines_test/0
expand_tabs_test/0            remove_duplicates/1
expand_template/2             remove_duplicates_test/0
extract_attribute/2           remove_leading_and_trailing_whitespace/1
extract_attribute_test/0      remove_leading_and_trailing_whitespace_test/0
extract_prefix/2              remove_leading_whitespace/1
fetch/2                       remove_prefix/2
fetch_test/0                  remove_prefix_test/0
file2lines/1                  remove_trailing_whitespace/1
file2lines_test/0             replace/3
file2md5/1                    root_dir/0
file2numberedlines/1          rpc/2
file2numberedlines_test/0     safe/1
file2paras/1                  show_loaded/1
file2stream/1                 signed_byte_to_hex_string/1
file2string/1                 signed_byte_to_hex_string_test/0
file2template/1               skip_blanks/1
file2term/1                   skip_blanks_test/0
file_size_and_type/1          skip_to_nl/1
find_src/1                    skip_to_nl_test/0
first/1                       sleep/1
flatten_io_list/1             spawn_monitor/3
flush_buffer/0                split_at_char/2
for/3                         split_at_char_test/0
force/1                       split_list/2
foreach_chunk_in_file/3       split_list_test/0
foreach_word_in_file/2        string2exprs/1
foreach_word_in_string/2      string2exprs_test/0
forever/0                     string2html/1
get_erl_section/2             string2latex/1
get_line/1                    string2lines/1
get_line/2                    string2lines_test/0
have_common_prefix/1          string2stream/1
have_common_prefix_test/0     string2stream_test/0
hex2bin/1                     string2template/1
hex2bin_test/0                string2template_test/0
hex2list/1                    string2term/1
hex2list_test/0               string2term_test/0
hex_nibble2int/1              string2toks/1
hex_nibble2int_test/0         string2toks_test/0
id/1                          sub_binary/3
include_dir/0                 template2file/3
include_file/1                term2file/2
interleave/2                  term2string/1
is_alphanum/1                 test/0
is_blank_line/1               test1_test/0
is_prefix/2                   test_function_over_substrings/2
is_prefix_test/0              tex2pdf/1
is_response_correct/3         time_fun/2
keep_alive/2                  time_stamp/0
lines2para/1                  to_lower/1
list2frequency_distribution/1 to_lower_test/0
list2frequency_distribution_tetrim/1
longest_common_prefix/1       trim_test/0
longest_common_prefix_test/0  unconsult/2
lookup/2                      unsigned_byte_to_hex_string/1
lorem/1                       unsigned_byte_to_hex_string_test/0
ls/1                          which/1
                              which_added/1

Now I find this very convenient when I write a new small utility function
I stick in in elib1_misc.erl – no mental anguish in choosing a module
name is involved.

The observation that I find this very-convenient is telling me something
about modules – I like my elib1_misc it feels right.

(aside – It seems many development projects have their own private
lib_miscs …)

Which brings me to the point of my question.

Do we need module’s at all? Erlang programs are composed of lots of small
functions, the only place where modules seem useful is to hide a letrec.

The classic example is fibonacci. We want to expose fib/1 but hide the
helper function fib/3. Using modules we say

-module(math).
-export([fib/1]).

fib(N) ->
    fib(N, 1, 0).

fib(N, A, B) when N < 2 -> A;
fib(N, A, B) -> fib(N-1, A+B, A).

The downside is we have had to *invent* one module name math – whose *only*
purpose is to hide the definition of fib/3 which we don’t want to be made
callable.

If we put a second function into the module math, then this second function
could call fib/3 which breaks the encapsulation of fib/3.

We could say:

let fib = fun(N) -> fib(N, 1, 0) end
in
   fib(N, A, B) when N < 2 -> A;
   fib(N, A, B) -> fib(N-1, A+B, A).
end.

I hardly dare suggest a syntax for this since I’ve been following
another thread in this forum where syntax discussion seem to encourage
much comment.

** Please do suggest alternative syntax’s here – but do not comment on
other peoples suggestions …

I would like to just talk about why we have modules.

Another question:

Does the idea of a module come from the idea that functions have to be
stored somewhere, so we store them in a file, and we slurp the
file (as a unit) into the system, so the file becomes a module?

If all the files were store by themselves in a database would this
change things.

I am thinking more and more that if would be nice to have *all* functions in
a key_value database with unique names.

lookup(foo,2) would get the definition foo foo/2 from a database.

The unique names bit is interesting – is this a good idea. Qualified
names (ie names like xxx:foo/2) or (a.b.c.foo/2) sounds like a good
idea but but when I’m programming I have to invent the xxx or the
a.b.c which is very difficult. It also involves the “decision problem”
if the namespaces xxx and a.b.c already exist I have to *choose* which
to put my new function in.

I think there might be a case for alises here joe:foo/2 could be used
while developing “joe” would expand to a horrible random local string the
real name being ab123aZwerasch123123_foo/2  but I would not be able to
publish my code or make it available to a third_part before I had
chosen a sensible name.

(( managing namespaces seems really tricky, a lot of peoople seem
to thing that the problem goes away by adding “.”http://groups.google.com/”s to the name
but managing a namespace with namees like foo.bar.baz.z is just as complex
as managing a namespace with names like foo_bar_baz_z or names like
0x3af312a78a3f1ae123 – the problem is that we have to go from a symbolic
name like www.a.b to a reference like 123.45.23.12 – but how do we discover
the initial name www.a.b? – there are two answers – a) we are given the name
(ie we click on a link) – we do not know the name but we search fo it ))

When programs are small we can live with “just the code” in “a few
modules” the ratio of code to meta data is high.

When programs are large we need a lot of meta-data to understand them.

I would like to see all functions with all meta-data in a data base.

I’d like to say:

   lookup(foo,2,Attribute) when Attribute =

      code|source|documentation|type signatures|revision history|authors|…

The more I think about it the more I think program development should
viewed as changing the state of a Key-Value database.

So I imagine:

    1) all functions have unique names
    2) there are no modules
    3) we discover the name of a function by searching metadata
       describing the function in a database
    4) all public functions (think open source) are in the same
       database

We could make a system to do this.

I think this would make open-source projects easier, since the
granularity of contribution goes down. You could contribute
a single function – not an entire application.

(( A problem with GUT style open source projects is there is
   not one database of functions, I often what one function from
   this project, another function from another project — the
   granularity of reusable parts should be the individual function.

   functions are really easy to reuse
   modules are more difficult to reuse
   entire applications are very difficult to reuse
     (Unless there are isolated through a communication channel))

Possible extensions.

    1) Voting for promotion
    2) A review process

Given a raw database will *all* functions in it – we could derive an
“approved” functions database.

Popular functions could be moved to the approved database – the
review process would need to be discussed – so kind of peer-review/wiki
stuff.

Comments?

Volunteers?

/Joe

to Joe Armstrong, Erlang

to Ralf P. Gans, Erlang

This is only true because that’s the implementation we have. The VM’s ability to maintain a couple of versions of a particular function could still be maintained on that granular a level..

… Provided there’s a way to, with atomicity, ensure that nothing calls the new version of a function without all of the related changes in associated functions rolling into the VM at the same time, since you may be moving quite a bit around if you find yourself refactoring a few functions to add new capabilities. And that IS something that is easier with modules as a complete unit of dependency.

Is there a way in a clean semantic manner track cross-function dependencies (potentially through some type of version tags?) in such a code database?

to Joe Armstrong, Erlang

Aren’t you just redefining the problem? Your next problem is going to be to name more generic type of functions. Imagine you have 3 different search functions which do different things in different modules. Your solution would instead of:

foo:search(), bar:search(), baz:search()

give

foo_search(), bar_search(), baz_search()

now you want to add a function that does replace; but replace what? say you call it

replace()

now I want to make one because it is specific to my problem so I have a decision problem

foo_replace() ? bar_replace() ?

so I will end up defining a prefix so that my versions of functions (where they collide with the already existing ones) always are unique thus:

mazen_replace()

now everyone starts creating functions with prefixes, now they want to start extending other functions so they build a prefix before the those:

mazen_foo_replace()

and before you know it you will have 20 functions ending with “search()” where search means what ever the prefix hints at…

So you are back to square -1 because now it is worse, you don’t have modules and you don’t have applications beacuse an application would become what modules was before it.

In fact, this is already a problem, all my opensource applications (and non-opensource as well) have a prefix on the modules because I can’t call all my server type modules “server” (there is no namespace).

My 2 cents.

/M

to Erlang Questions

>
> So I imagine:
>
> 1) all functions have unique names
> 2) there are no modules
> 3) we discover the name of a function by searching metadata
> describing the function in a database
> 4) all public functions (think open source) are in the same
> database
>
> We could make a system to do this.

Hm… you’d still need *some* kind of modules.

For instance, in Webmachine each resource is expected to have some, or all, of predefined public/exported functions: to_html, resource_exists, allowed_methods etc.

So, each resource will have functions with the same name. If we move all functions to a global namespace, then how do we differentiate between these functions? Probably by naming them resource_name_function_name?

Furthermore, what do we do with behaviours? A behaviour is encapsulated within a module, also with a predefined set of exported funcions (and any number of custom exported functions).

to Erlang

There are two good things about modules:
* they are a higher level of abstraction
* they support good working practices

Lets take a module that I use a little bit – and which a colleague
introduced me to:
http://erldocs.com/R14B02/stdlib/digraph.html?i=2&search=digra#get_digraph/1

I can use the couple of exposed functions of that that I know,
confident that if it comes to it I can come back and master the rest
of the interface later. I have confidence that there is a rubustness
there that makes code using it maintainable. (my idea of maintainable
is to be able to ignore most stuff, most of the time…). I can learn
about the code at this higher level of abstraction.

We too have a module with loads of random functions in it, well we
have three (hn_util, util, util2), and each of them is just library
functions. In the utility modules the ratio of exported to
not-exported fns is maybe 7 to 1.

In functional modules the exported/not ratio goes from 1-1 (in modules
that define apis) to 1-3 against.

If we had the (promised) -export-to() directive the number of pure
exported fns would drop.

Given that we build our testing strategy around -export() this
separation is critical…

Gordon

—
Gordon Guthrie
CEO hypernumbers

http://hypernumbers.com
t: hypernumbers
+44 7776 251669

to Erlang Questions

to Erlang

…and also…

following on from what Dimitri said while I was writing me e-mail, we
hang unit test onto modules and design them at that level of
abstraction

to Joe Armstrong, Erlang

Modules have many uses, but they are often mixed up. Yesterday I made this drawing:

to Joe Armstrong, Erlang

Hi,

to Joe Armstrong, Erlang

Sorry, my mail got strangled in flight. Here we go again… in plain text.

The gen_server module is used for three different things:

– as a collection of API function (left)
– a place to put the code that controls the gen_server process (middle)
– a place to describe the abstract notion of a gen_server behavior (right).

Ideally, I think those three usages should be separate language constructs, but probably not in separate files (or other kinds of units).

When using a gen_server, it would be nice if you did not have to expose the behavior api to the “user” of your module.

Currently, the module is the unit of code loading. I think we could use a bette mechanism, perhaps one that supports an STM-like thing in the VM to control the loading of a set of modules. I don’t have much deep experience with code loading in Erlang (but do have it on Java), and it just seems to me that being able to control atomic loading of .. say a new version of an Erlang application would be good to have supported in the VM.

Gilad Bracha (who’s working on Newspeak) has an excellent blog on new notions of modues that you should read. You can start here http://gbracha.blogspot.com/2009/06/ban-on-imports.html

One of the things he’s arguing is that modules should be VALUES. A “module” in newspeak is kind of a template (function), which take other module instances as arguments. So you hook up a system by applying such templates (possibly using letrec).

If we had a visual programming environment, then many things don’t need a name. Self is an example of this (Self, the video: http://video.google.com/videoplay?docid=5776880551404953752#)

Kresten

to erlang-q…@erlang.org

On 05/24/2011 04:49 PM, Josh Johnston wrote:
>
>
> In a normal language this would be a problem, but in erlang we can get
> the desired behaviour by using pattern matching, right?
>
> So instead of
>
>
> to_html()
> resource_exists()
> etc
>
>
> We have
>
> resource(to_html, …)
> resource(exists, …)
> etc
>
> I’m not disagreeing with you – we do need some good way to do this. But
> I don’t think modules are necessary in this case.
>

Pattern matching might work here, but do away with modules and using
pattern matching would a heavier load on the programmer.

Modules or not, it does not have much effect on the VM, we can
just make the VM work without modules. But I think programmers
would have to work harder, distinguishing which function is which.

Although I don’t really like the Erlang module, but I think it’s
better with it than without.

Xiaopong

to Kresten Krab Thorup, Erlang

On Tue, May 24, 2011 at 11:05, Kresten Krab Thorup <kr…@trifork.com> wrote:
>
> One of the things he’s arguing is that modules should be VALUES.  A “module” in newspeak is kind of a template (function), which take other module instances as arguments.  So you hook up a system by applying such templates (possibly using letrec).
>

This is a great idea. Why? Because it is the path of existential types :-)

Notice that in Erlang, a module is a Value. It is possible to call
Mod:Fun(..). Second, notice that the parameterized module extension
allows us to parameterize a module over some other module.
Specifically we could have

-module(RBTree, [OrderMod]).

where OrderMod is a module which has a function

-opaque t() :: …..
-export_type([t/0]).
-spec order(t(), t()) -> lt | eq | gt.

and so we have generalized our RedBlack trees to be able to use any
ordering relation, not simply the one that is built it.

The style Gilad is advocating is the “Fully functorial style” from the
Standard ML community. A Functor[1] in SML terminology is a function
from module to module (“functor” stems from category theory where it
designates a “function”[2] one level up). In the style, you build up
programs by stitching together smaller modularized parts building
larger and larger modules in the process.

Example: The gen_server is a functor from a module (which obeys the
gen_server behaviour and has handle_* functions) to a specific
instance of a server.

It also gives you quite the path towards the concept of OOP, as can be
observed from the parameterized module extension and its uses in
various places.

In languages with static types, you can of course enforce that functor
applications are type-correct. If you allow modules as values in such
languages you get existential types which is a way to get something
OOP-like into those languages[3].

So after a little tour around: Erlang already has this :P

Modules are an essential part of building large systems. The very
notion that you can pack up functions in a module and ship them as a
unit is quite important. Also the notion you can replace a
module-component at once. I feel that the problem of function
placement in the right module is less of a problem compared to the
hell it would be with one large global namespace. And it is much more
than just a “bunch of functions thrown together”.

Specifically, modules allows me to have an *abstract* view of a data
structure. In the above, I have no idea what t() is. I am only allowed
to compare two t()’s to each other with OrderMod:compare(T1, T2). That
is, the only algebraic operation I am allowed to do is this. Other
parts of the program may have a different view of t() and be able to
do more with it, but to me, it is an opaque term. It matters because
someone can go replace t(), and my code does not have to be changed at
all. This is impossible if t() leaks. And I am grateful the dialyzer
can find places where it is leaking.

Likewise, a module can hide the fact it is implemented with a
gen_server. So I can go replace it with a gen_fsm, and nobody has to
alter their code because the API i exported was exactly the API I will
keep stable.

Processes componentize the heap.
Modules componentize the code.

[1] Beware of connotating the word functor in programming with
something specific in computer science. The meaning is different in
Standard ML, in Haskell and in C++.

[2] I am lying. The basic notion is a morphism of which the function
is a special case when considering the category of Sets.

[3] I still don’t get why you can’t use static typing in Erlang. The
conservative approach is simply to use polymorphic variants for all
message passing and it would essentially work right out of the bat.

—
J.

to Erlang

Hi all,

I am sticking to the request to not comment on other solutions but
present my own thoughts…

1. Some units of code only make sense as a collection. who would use
part of gen_server in isolation? Well, you could,but to have it
separate exposes it to modification that could distort the focused
purpose of the unit.

2. some functions have intentionally hidden aspects, private calls etc
and these again are best left hidden or considered as a logical whole
otherwise side effects or bloat may occur.

3. I see no reason why we cannot logically group functions into
modules but not have the code in there…i.e. the module contains the
references to functions that make it up. this allows logical handling
of groups of functions for code management and can allow one function
to appear in multiple modules without code duplication. it also allows
one to add/delete/replace a function in a module knowing that the
source code for the other functions have not been touched!

4. naming conventions should be given great thought for simplicity and
intuitiveness. functions could be in n sets, so a strict hierarchy may
not suit.

5. if functions are named as being in modules, the need to provide
abstraction to the full name may be advantageous, so modulename.sort/3
can be used and the module definition translates this to a unique
longhand which can change over time. The programmer should also be
able to use the longhand at will using the module as a convenient bulk
import/compile vehicle that will break at compile time (one would
hope). Some form of ‘expects…’ notation could allow a form of
protection against inadvertent swapping of functions in shared modules
altering behaviour.

6. we may want 3 elements to naming of functions: shorthand, longhand,
and version, so one can use the latest version of a longhand named
function, lock to a specific version of a longhand named function or
use the shorthand latest version or other such combinations.

7. n- tier modules – modules of modules? possibly.

Tim

Tim Carpenter
Amphibian Ltd
London
ENGLAND
+44(0)7900 888826
amphib…@gmail.com

to Erlang

Greetings,

It would be very nice to have module renaming/namespacing that allowed
me to do the following:
The modules “misc_lib” and “misc_lib_helper” from developer A should be
able to call each other with their original names.
The modules “misc_lib” and “misc_lib_helper” from developer B should be
able to call each other with their original names.
I should be able to call all 4 at the same time with names that I have
create/renamed.

So, I need to load modules as a set/application/??? that renames them
outside of that set (it would be ok to have the same prefix added to all
of them), and still allows the files in the set/application/??? to keep
using to the old names.

When it comes to improving stdlibs I suggest that we create new, better
modules. Keep the old ones (for ever?) but print a big “deprecated” all
over the man pages.

bengt

> I think we need modules, but this post sparked a thought I have had
> for some time now.
>
> I’d like to have module renaming/namespacing!
>
> Specifically, I want a mechanism by which I can clean up the naming
> mess in the stdlib without affecting old programs. That is, I want to
> be able to, at an application/module level, rename function calls in a
> way such a way that can clean up parameter orders and so on. I don’t
> care about how the solution is. I am after the goal: Cleaning up the
> stdlib mess.
>
> You may argue this is a messy thing, but I think it is a necessity if
> you want to move the standard library on:
>
> * We can’t alter existing functions because people rely on them.
> * Existing functions are not consistent.
>
> This is a social deadlock I want to break. In fact it is a deadlock I
> think we can only break if we allow the coexistence of more than one
> module with the same name (but stemming from different namespaces). I
> don’t want nesting. Just two levels, so I can say
>
> -use_namespace(v2_modules).
>
> in modules which use version two of the stdlib in which things are
> sensibly named. At some point you can then flip around and compile old
> code with a compatibility layer.

to Joe Armstrong, Erlang

I am working on state-oriented programming model, and I have the same question as you.
In the view of SOP, erlang process is a FSM, and suitable granularity of functions as states.
The finite set of states of a FSM just like erlang module, but it always cross modules boundaries.
So in SOP model, the module concept has been very vague, just for unique distinct name of function.
The model of a state activity is a function as bellow:
act(Entity, Input) -> {ok, Output} | {ok, Output, UpdatedEntity} | {error, Error} | {stop, Output, Entity}
The output of the state may be {NextState, Output}.
The NextState may be the next state activity function or just a directive of next state.
Then FSM engine matches the directive to a state activity then transfers the previous state to the new state with the previous output as input.
The state activity function may be from anywhere, function in a module, binary loaded from database, code dynamically generated by JIT.

——–
Gary Hai

to Joe Armstrong, Erlang

On Tue, May 24, 2011 at 10:06, Joe Armstrong <erl…@gmail.com> wrote:

> Why do we need modules at all?

I think we need modules, but this post sparked a thought I have had
for some time now.

-use_namespace(v2_modules).

—
J.

to Jesper Louis Andersen, Erlang

My 2 pennies worth: I don’t think getting rid of modules is a good
plan. Having global (mangled) names is a path to misery. In the
languages that do mess with names (python, C++, etc) it creates a
bloody nightmare. I would steer clear of this. I like the idea of
having a global database of code, but not all of it floating in some
etherial top level namespace.

Most of what’s cool about http://www.haskell.org/hoogle/ and
http://hackage.haskell.org/packages/hackage.html is made possible by
the combination of good standardised build tools and hosting
repositories, so I’d be more included to focus on improving (and
extending) things like rebar and agner/erlware to provide the same
kind of developer-experience without changing the language so much.

Introducing letrec is a bloody marvelous suggestion. Also making it
easier to do partial application and function composition would
simplify a lot of the re-use cases without messing with the module
system.

I’m not going to stray off too far into alternative-solution land, but
given the OTP team reads this list, I don’t want to miss the
opportunity to mention some minor things about the module system that
could do with an overhaul.

– Fully supported parameterised modules

Not because I want to use them as a state bag, but rather that they
make it easy to implement something akin to a functor without having
to reside in a process. This means that all the tools need to support
this concept and it is documented, etc. Extending the module system so
that you can -extend a module to create a new concrete implementation
would be useful:

%% finder.erl
-module(finder).
-functor([M]).
-compile(export_all).

find() -> walk(fun M:match/3).

%% rx_finder.erl
-module(rx_finder).
-extends(finder, [re]).

Perhaps the syntax/approach there isn’t quite right, but the point I’m
getting at is that what with parameterised modules and -extends, you
should be able to compose stuff easily.

– Better support for *import*

As the quantity of code you’re dealing with increases, you need to be
able to import discrete units of it in simpler and more intuitive
ways. I’m not exactly a fan of Java, but you can see in that world
that the quantity of libraries (often open source ones) being used on
any project means you simply cannot avoid having to import things
(packages, classes, static methods/functions) without going mad.
Although Erlang is still (thankfully) simple enough not to have this
problem, I suspect as the number of projects blossoms it will
eventually approach the critical mass.

Also being able to support something like -import_alias(name, mod)
would be quite handy, although I realise this is fairly trivial with a
parse_transform.

– Fully supporting packages

Personally I think the __ convention that most
of us follow is fine, but it does get rather tedious. Package,
namespaces or nested modules are all solution that are known to work
and are widely adopted by other languages. The package system in
erlang works fine, but not all the tools properly support it today
(cover, reltool and a few others are broken).

to bengt….@ericsson.com, Erlang

On 24 May 2011 11:39, Bengt Kleberg <bengt….@ericsson.com> wrote:
> Greetings,
>
> It would be very nice to have module renaming/namespacing that allowed
> me to do the following:
> The modules “misc_lib” and “misc_lib_helper” from developer A should be
> able to call each other with their original names.
> The modules “misc_lib” and “misc_lib_helper” from developer B should be
> able to call each other with their original names.
> I should be able to call all 4 at the same time with names that I have
> create/renamed.
>
> So, I need to load modules as a set/application/??? that renames them
> outside of that set (it would be ok to have the same prefix added to all
> of them), and still allows the files in the set/application/??? to keep
> using to the old names.
>
>

I was thinking about this in terms of some kind of special -import or
-alias directive. I think it’s possible to do just with a
parse_transform, as long as the set can be fully determined at compile
time. That last point though – therein lies the rub. App-upgrades
change things, and those things can include modules. Dragons.
Nuff-said.

to Tim Watson, Erlang

On 2011-05-24, at 13:19 , Tim Watson wrote:
> In the languages that do mess with names (python […]

As far as I know, Python never “messes with names” unless asked explicitly via the “__” prefix, which is rarely used and generally only there to avoid naming conflicts in inheritance cases.

Do you have instances, besides this quite explicit one, where Python performs any name mangling?

to Jesper Louis Andersen, Erlang

to Jesper Louis Andersen, Erlang

to Tim Watson, Erlang

to Joe Armstrong, Erlang

> Now I find this very convenient when I write a new small utility function
> I stick in in elib1_misc.erl – no mental anguish in choosing a module
> name is involved.

But it’s not very helpful to someone else who’s going to have to
mentally parse your lib_misc library to figure out what they’re
looking for. It does sound a bit like a dumping ground.

Perhaps if those libraries were broken into their constituent parts
and then those into their constituent projects, they would be easier
to re-use. I see the same thing on many projects – a_config.erl that wraps a gen_server with maybe some
calls to application:get_env and a _log.erl and so on. I
nearly wrote a erl-config package on github until I realised gproc
would be a better choice for my needs.

I think that’s an oversimplification. Compilation units (e.g., putting
code in different files) is beneficial for humans first and foremost.
I think that categorising functions based on subject-area and/or the
process they relate too is very useful. Once you do that categorising,
putting all the categories in once place is very sensible. OCaml and
Haskell programmers (of which I’m only experiences with the former),
use letrec (and similar concepts such as Haskell’s `where`) heavily,
but would not dream of doing away with our module systems just because
we can encapsulate things in this way.

Modules are a useful categorisation for the human mind to remember
things by. Like packages, they help us remember where to look for
things. They’re also useful for hiding – breaking encapsulation is a
choice, not something you cannot help when designing a module.

Actually if you put all your mathematical functions in this module, it
would be more useful (and easier to remember).

Not sure that’s such a terrible thing really.

Having letrec (or where) is an awesome idea. Don’t actually care much
about the choice of syntax – just another one to learn.

Probably, yes. It is convenient and familiar to do so.

Not really, as long as the database is always available and I *never
ever* run into a situation where I can’t work because I’m not
connected.

I love the idea of a massive code repo/server, but having everything
in one namespace sounds like a nightmare.

Still I think it’s the right thing to do.

Yuk. ;)

You’re right it is hard, but it is possible.

I like the idea of finding a function based on a wider range of
attributes such as these, but not choosing the route of dispatch based
on a loose predicate. Lookup/3 would have to guarantee that it will
resolve to a single callee every time, which I can’t see being
possible.

to Joe Armstrong, Erlang

> Where are two interpretations
>
> a)  “when you *run* this program and find a call to foo:bar/2 go and see if
> you can find the code
> for foo, load it then call the function bar/2″
>
> b ) “When you *compile* this program find the code for foo:bar/2 and
> extract the code for bar/2 and include it in my code” – in this case you
> resolve the name at compile time.
>
> I guess when developing a) is useful but when you deploy code it should be
> b)
>

I completely agree. I’m also quite willing to loose the convenience of
(a) providing the compiler is fast enough.

to Erlang

Very strange topic for me.

I’d like to know if there will be hierarchial modules in Erlang,
because tree of packages is a rather good idea:

erlyvideo.mpegts.encoder
erlyvideo.rtp.encoder

But plain module namespace is also ok. It would be impossible for me
to work with 30K LOC with plain function namespace.

to Max Lapshin, Erlang

> But plain module namespace is also ok. It would be impossible for me
> to work with 30K LOC with plain function namespace.

I think what Joe is getting at – he’ll correct me if I’m wrong about
this – is to refer to functions not only by name, but by other
metadata as well. I’m not sure what this looks like empirically, as
like you I can’t see how using only function names you could work with
a flat namespace. Looking up functions by other (additional) metadata
such as type signature, author, date, company, category (mutiples?),
pre/post conditions, safe guarantees, etc – that would work. In fact
if you think about a process engine (such as BPMS solutions and the
like) then process (i.e., function) is often selected dynamically
based on many attributes, not just name.

Just how this will work in practise as a replacement for existing
(shall we say normal) way of referring to a function that is in a
module, I could not say. But it is an interesting concept. Perhaps
using a URI as the name and thereby being able to have multiple URIs
representing the various different category-based paths to identify
the function – the “aliases” could then redirect to the direct,
uniquely named resource which would in turn return the function
metadata (and binary code presumably).

to Max Lapshin, Erlang

to Tim Watson, Erlang

to Erlang

Greetings,

Perhaps it is possible to do parse_transform when loading, but I have
only heard about this at compile time. I meant a way to load modules,
not compile them.

bengt

to Joe Armstrong, Erlang

The problem of Erlang with regards to namespaces and modules goes further than just module names. In Erlang, the following units can be sharing/clashing namespaces: the process registry, ETS tables, module names, application names, shared records, etc. They’re currently entirely disjoint and only held together by convention.

However, any change to one of the solutions needs to somehow be reflected into the others, even if it’s still by convention.

IMO the basic hierarchy of Erlang seems to begin with the OTP release. The name of an OTP application will decide the prefixes used in module names, process names, table names, shared record names, etc. There is nothing to check that application names are unique except the programmer looking out for that, while xref and the compiler can help a bit with the rest.

I think we can generalise OTP to have this given kind of organisation:
release -> OTP applications -> [Modules, Tables, Shared Records, Processes]

An application itself wouldn’t have to worry about namespaces (if we forget included applications). The namespace problem comes at the release stage or if we’re developing entirely outside of OTP, still using applications or their equivalent.

To me, this makes it feel like namespace management/giving aliases and replacement of values should be managed from the release point of view, but I can’t offer any suggestions as to how this could take place without relying on file renaming and strict use of ?NAMESPACE-like macros throughout the code. Nothing easily backwards compatible, and nothing to help people who are not using releases though.

This in no way would solve Joe’s problem with being undecided as to where functions should go, but I think getting rid of modules altogether in Erlang wouldn’t be doable because of the other dependencies. In places like Javascript (client-side) or schemes without good module functionality, you pretty much just include the files and people then tend to wrap everything in functions or objects to provide an ad-hoc equivalent to namespaces or modules. I can see this behaviour emerging if Erlang were to go that way.

—
Fred Hébert
http://www.erlang-solutions.com

to erlang-questions

2011/5/24, Joe Armstrong <erl…@gmail.com>:
[…]

This only works if the function in itself useful. Somehow I don’t
think this is the case the most time. If I need not a function, but a
functionality, there’s a chance there is an ets table behind or a
process, some configuration file setup – then the whole “import a
single function” doesn’t help, but just adds complications that even
the last helper function has to have a globally unique name…

to Frédéric Trottier-Hébert, Erlang

to Joe Armstrong, Erlang

(am sending again as this did not seem to get through)

Hi all,

Tim

_______________________________________________

to Joe Armstrong, Erlang

> Yes – how this would work I don’t know – my Haskell friends say that
> searching for
> function by type signatures is good.

I’ve only used OCaml commercially, but what little Haskell I’ve played
with this is a very useful feature (hoogle and hackage combined with
cabal) for finding and using stuff.

Not quite. At point (3) you probably get to github, bitbucket or some
such. At that point, you *may* be able to reuse the library and/or
application code in your project, in which case you install it with
epm/sutro/agner or pull it in as a dependency using rebar. Only when
the code is hidden away in the module and not reusable do you have to
copy-paste – I think it’s this (latter) case that you’re not happy
with and I agree it’s poor show.

By way of example, I’m working on a node/cluster monitoring tool
(primarily a web application with web sockets notifications). I built
the library code (for monitoring) by re-using the eper (performance
monitoring) library and a couple of other things (zero-conf for
network discovery and whatnot). I built the web application on
misultin for its websocket support. I also have some common (to the
project) library code which uses the esl/parse_trans library from
github to support better use of records and things like that. For the
database application, I used the esl/setup library from github to
separate the mnesia database/schema setup from the general purpose use
cases and to generate (run once) setup scripts and the like. None of
this is copy and paste re-use.

I think it would be worth distinguishing between two disparate concerns here.

1. Where does the (binary or other representation of AST) code come from
2. Where does the/any (runtime) state reside

I don’t always want to RPC just to run a library routine, especially
if that call is happing frequently in a tight “loop” and the code
needs to run fast. Much as I love Erlang’s location transparency, one
cannot just assume that the network isn’t there.

Things I think are potentially awesome here are:

1. Importing from a URI
2. Having the binary code in a repository that is searchable, versioned, etc

In fact this is hardly an unfamiliar model – consider couchdb and/or
riak. When you define a map-reduce function as say javascript, you
distribute/upload this function definition to the server for later
use. I think my two major concerns (that are by no means
unsurmountable) are:

1. wanting to be able to work when I’m not connected to the code server
2. wanting to be able to define the import based on more than just the name

Point (2) is telling. In your example, you imported from
http://a.b.c/modname – you still haven’t gotten rid of the module. I
know it was a flippant example and besides the point. Question is, can
you define the search in such a way as to determine exactly where the
dispatch must go? The Haskell folks say searching by type is immensely
useful, and they’re right. But they don’t compile based on this – they
find what they’re looking for and then reference it (explicitly) by
name.

to bengt….@ericsson.com, Erlang

> Perhaps it is possible to do parse_transform when loading, but I have
> only heard about this at compile time. I meant a way to load modules,
> not compile them.
>

You could hack something together to do it at runtime if you’re
importing/loading from an AST rather than a blob. I take your point
though – I did misunderstand slightly.

to Tim Watson, Erlang

I don’t understand what are you speaking about.

How do you imagine this magic function namespace in terms of plain files?

to Max Lapshin, Erlang

On 24 May 2011 14:38, Max Lapshin <max.l…@gmail.com> wrote:
> I don’t understand what are you speaking about.
>
> How do you imagine this magic function namespace in terms of plain files?
>

You forget about files. You’re code editor (IDE) is now a browser
based application. You search for things using a sidebar that
ultimately constructs a query to the code server to look for things
based on name/version/type-signature/author/safety/etc which becomes a
URL. You import things based on URLs. Your code is compiled and hosted
on the server. The local file system doesn’t have to come into it per
se.

This last point is partly one of my concerns though. I don’t always
have an internet connection and something that doesn’t work without
one is potentially going to cause me problems. Perhaps if it was easy
for me to copy the entire repository and run it locally, synchronising
when I’m reconnected – that would do nicely. Then the service would be
providing something akin to source version control as well.

My head now hurts. :)

to Max Lapshin, Erlang

When you need to call a library function you query the database it helps you find the function
you need from the data in the database – you never know the real name of the function or anything about
the namespace it’s in.

The problem with plain files is that they become unmanageable when there are lots of them.

A few years ago I had problem with storage – disks weren’t big enough – now I have TBs of
raid storage and the next problem arises – how to find stuff. Finding my own code is difficult –
it’s just a matter of searching 43 K erlang modules on my local disk – finding other peoples
code is worse – Google and “ask a friend” is the best solution I know of.

As systems get very large the notion of files and modules seems to break down, to be
replaced by “search-able stuff in a database” – I’m suggesting that the smallest unit that
should be searchable/reusable/discoverable should be the function. And that to make it
searchable we need to add a lot of meta data to the functions.

to Max Lapshin, Erlang

This conversation is reminiscent of the teapot/teacup architecture from TCL 8.4

http://wiki.tcl.tk/7579

The description of that architecture may be on target?

On May 24, 2011, at 9:10 AM, Max Lapshin wrote:

> I can’t imagine myself working with system, that uses non-plain files
> for storing source code.
> Nobody will fix for me Textmate, git or ack to work with this database.
>
> Modules give your clear structure of functions. Without them you will
> end with mpegts_encode, mpegts_decode,
> and will break your head when there will be 3 or 4 underscores in name.

Thanks,
Barry

to Tim Carpenter, Erlang

to Max Lapshin, Erlang

or should the name be encode_mpegts and decode_mpegts the problem with structured names
is choosing the name – not the fact there is structure in the name.

Imagine the www was just one big structured name and there were no search engines
how would you ever find anything?

/Joe

to Joe Armstrong, Erlang

On Tue, May 24, 2011 at 4:22 PM, Joe Armstrong <erl…@gmail.com> wrote:
>

> Why not? – a plain text file is an abstraction – in reality its a collection
> of magnetised zones
> on a disk, or electron spin states or something, this gets turned into a
> collection of
> bits, the bits are rendered in a editor. Plain text doesn’t exist – it’s
> just an abstraction.
>

How about storing the code in the old school plain text files but with
additional, also plain text, index/metadata files ?
That would satisfy both the searchability and legacy compatibility

—
cyryl

to Joe Armstrong, Erlang

> On Tue, May 24, 2011 at 1:56 PM, Max Lapshin <max.l…@gmail.com> wrote:
> Very strange topic for me.
>
> I’d like to know if there will be hierarchial modules in Erlang,
> because tree of packages is a rather good idea:
>
> No it’s not – this has been the subject of long and heated discussion and is
> why packages are NOT in Erlang – many people – myself included – dislike
> the idea of hierarchical namespaces. The *dot* in the name has no semantics
> it’s just a separator. The name could equally well be encoders.mpg.erlyvideo
> or mpg.applications.erlvideo.encoder – there is no logical way to organise the
> package name and it does not scale –

packages are NOT in Erlang? Then the related code should be removed because
erl.lang.number:plus(1, 1) definitely works.

Also, I think the Haskell guys would disagree about packages not scaling.

—
Anthony Ramine
Dev:Extend
http://dev-extend.eu

to Max Lapshin, Erlang

> I can’t imagine myself working with system, that uses non-plain files
> for storing source code.
> Nobody will fix for me Textmate, git or ack to work with this database.

I remember when Zope worked like this. I hated it, and to my
knowledge, none of the editors picked up support for it despite it
being pretty popular at the time. I believe they later changed to
uisng the normal filesystem, but it’s been a long time since I paid
any attention to that project.

jack.

to Joe Armstrong, Erlang

to Anthony Ramine, Erlang

On Tue, May 24, 2011 at 16:31, Anthony Ramine <n…@dev-extend.eu> wrote:
>
> packages are NOT in Erlang? Then the related code should be removed because
> erl.lang.number:plus(1, 1) definitely works.
>

The problem is that a package hierarchy enforces a tree-like
structure. And we may not want the tree. You might want the same
module to reside in multiple packages in the hierarchy at the same
time. We are past 2005 now and we know what a tag cloud is. About time
we began thinking of how that would fit into a programming lang. :-)

The only real advantage of a package tree is that you can:

* Shove some packages underneath a common notifier. This limits the
packages currently in scope.
* Shortcut reference to package if they nest.

I think both would be better served by a system different from hierarchy.

—
J.

to Joe Armstrong, Erlang